Interactive explainer

Anatomy of an RS Vision-Language Model

Every RS-VLM is three design choices. Learn the pipeline once, then read papers as diffs.
01 · Mental model

Three slots, every paper

File every paper into these three slots and the literature stops feeling like a zoo. When you read a new RS-VLM paper, ask only: what changed in each slot compared with LLaVA-1.5?

A
Slot A · Vision encoder

What encodes the pixels?

The vision encoder turns the image into a grid of feature vectors. Nearly everyone uses a CLIP/SigLIP ViT, which is exactly the field's weakness (CLIP-blindness: counting, relative position, small objects). The frontier move is swapping in an EO-native encoder.

CLIP ViT-L (default)SigLIPDINOv2GeoFM encoder (rare, G11)multi-encoder (EarthMind)
B
Slot B · Bridge

What bridges to the LLM?

The bridge (projector) maps vision features into the LLM's token embedding space, so image patches become "words" the LLM can attend to. Simple MLPs won; query-based compressors trade fidelity for fewer tokens.

2-layer MLP (LLaVA default)Q-Former / resamplerpixel-shuffle downsampleper-sensor projectors
C
Slot C · Grounding

What grounds the output?

How does language connect back to space? This is where RS-VLMs genuinely innovate: from coordinates written as plain text, to special tokens that trigger a mask decoder, to agentic zoom loops.

none (caption/VQA)bbox-as-text (GeoChat)<SEG> → mask decoder (GeoPixel, TerraScope)attention prompting (EarthMind)agentic search (GeoViS/GeoEyes)
02 · Interactive

Pick a model, watch what changes

Each model is drawn as a diff against the LLaVA-1.5 baseline. Shapes assume a 336×336 RGB input unless noted.

Same as LLaVA-1.5Differs from the baseline
03 · Deep-dives

Inside each stage

Open the stage you're stuck on. The first one is open to start.

Slot APatchify & the vision encoder
A ViT splits the image into 14×14-px patches: 336/14 = 24 → 24×24 = 576 patch tokens, each projected to a 1024-d vector, giving [B, 576, 1024]. Positional embeddings add location; ~24 transformer layers mix information globally.

Why this matters for RS: a 10k×10k scene resized to 336px makes a car smaller than one patch — it simply ceases to exist for the model. Fixes: AnyRes tiling (encode crops separately, concat tokens), or agentic zooming. Also note CLIP was contrastively trained to match captions — it keeps what is in the image, discards where and how many. That bias is inherited by every downstream RS-VLM (measured by RSMMVP).
Slot BThe projector — the entire "alignment" trick
The LLM only understands vectors in its own embedding space (e.g. 4096-d for a 7B LLaMA). The projector is often just:
vision_feats [B, 576, 1024]
  → Linear(1024→4096) → GELU → Linear(4096→4096)
  → visual tokens [B, 576, 4096]   # now "words"
These are spliced into the prompt: <sys> USER: <img₁…img₅₇₆> What is in the harbor? ASSISTANT:. Training Stage 1 freezes everything except this projector — hours, not days, of GPU time. That such a small module suffices tells you CLIP features and LLM word embeddings are already nearly linearly alignable — the deepest single fact in VLM land.
Slot CGrounding mechanisms — compared
Bbox-as-text (GeoChat): the model literally writes <obb>{47, 112, 98, 163, 30°}</obb>. Zero extra architecture; grounding is "just" next-token prediction. Fragile — numbers drift after further fine-tuning, precision limited by tokenizer granularity.

Special token + decoder (GeoPixel, LISA-style): the LLM emits a <SEG> token; its final hidden state becomes the prompt embedding for a SAM-style mask decoder → true pixel mask. Precise, adds a decoder to train, needs mask-annotated dialogue data.

Attention prompting (EarthMind): reweights LLM attention toward spatially relevant visual tokens; multi-granular (image→region→pixel) without coordinates in the text stream.

Agentic (GeoViS/GeoEyes): grounding becomes a policy — crop, look, decide whether to zoom again. Handles 10k×10k scenes; costs multiple forward passes per answer.
TrainingThe three training stages
Stage 1 — Alignment: freeze encoder + LLM, train projector on image-caption pairs (RS: SkyScript, RS5M, LHRS-Align). Teaches the LLM to "see" but not to converse about it.

Stage 2 — Instruction tuning: unfreeze LLM (or LoRA r=16–64), train on multi-task conversations (VQA, grounding, captioning mixed). This is where capabilities come from — and where RS models differentiate: GeoChat's 318k RS instructions, EarthDial's 11M+, MMRS-1M's multi-sensor set. Data construction is the real contribution of most RS-VLM papers.

Stage 3 (optional): mask-decoder training (GeoPixel), preference/DPO for robustness (RemoteShield), RL for search policies (GeoViS).
Gap G12Why multispectral input is hard for VLMs
CLIP expects 3 channels; Sentinel-2 has 13. Most "multi-sensor" RS-VLMs quietly render SAR/MSI to pseudo-RGB composites before encoding — throwing away SWIR, red-edge, and phase information. Real fixes mirror pure-vision GeoFMs: band-adapted patch embeddings (EarthDial-style), per-modality encoders fused before the LLM (EarthMind), or a wavelength-conditioned encoder (DOFA) swapped in as Slot A. Check a model's preprocessing config before believing its sensor claims.
04 · Check yourself

Four questions to test the model

If you can answer these without scrolling up, the three-slot model has stuck.

Q1
A 7B RS-VLM writes bounding boxes as text. After you LoRA-fine-tune it on your captioning data, grounding degrades but captions improve. Which slot broke, and why?
Q2
You need referring segmentation ("mask the flooded fields west of the bridge") on 8k×8k imagery. Which Slot C mechanism and which resolution strategy do you pick, and what data do you need?
Q3
Your model answers "how many aircraft?" wrong 70% of the time but describes airports fluently. Explain using Slot A. What experiment isolates the encoder as the cause?
Q4
Sketch (pseudocode) the forward pass of GeoPixel from image to mask. Where does the LLM hidden state meet the mask decoder?