Research map / Vision-language and multimodal / Visual question answering
TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TEOChat, the first vision-language assistant for temporal Earth observation image sequences, by adapting Video-LLaVA and training on the new TEOChatlas instruction dataset.
Problem
Existing vision-language assistants for Earth observation handle only single images and cannot reason over temporal sequences of EO imagery for tasks like change detection and damage assessment.
Approach
Adapt the Video-LLaVA architecture to EO image sequences with a temporally-shared CLIP image encoder, interleave image references between visual tokens, and instruction-tune on TEOChatlas, a dataset of temporal EO tasks built from fMoW, xBD, S2Looking and QFabric plus GeoChat_Instruct single-image data.
Key results
- TEOChat achieves 75.1% on fMoW RGB and 50.0 F1 on xBD damage classification, outperforming GeoChat (59.2%, 11.8 F1) and Video-LLaVA.
- TEOChat outperforms GPT-4o by 11.7 F1 on xBD damage classification and by 14.2 F1 on S2Looking change detection.
- TEOChat achieves 85.6 on zero-shot ABCD and 47.2 on zero-shot CDVQA, versus Video-LLaVA's 50.0 and 29.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train on single-image and temporal tasks rather than temporal tasks only (ablation) | TEOChat · fMoW RGB | accuracy: 73.5 -> 75.1 | Helped |
| fine-tune on TEOChatlas (ablation) | Video-LLaVA initialization · fMoW RGB | accuracy: 16.6 -> 63.2 | Helped |
| use LLaVA instead of Video-LLaVA initialization (ablation) | TEOChat · fMoW RGB | accuracy: 69.2 -> 66.8 | Hurt |
| use a SkyScript-pretrained image encoder instead of CLIP (ablation) | TEOChat · fMoW RGB | accuracy: 69.2 -> 55.6 | Hurt |
| fine-tune rather than freeze the projector (ablation) | TEOChat · fMoW RGB | accuracy: 69.2 -> 71.7 | Helped |
| include image references between visual tokens (ablation) | TEOChat · fMoW RGB | accuracy: 63.2 -> 69.2 | Helped |
| train for 14k rather than 7k iterations (ablation) | TEOChat · fMoW RGB | accuracy: 71.7 -> 75.1 | Helped |
| use TEOChat rather than GeoChat | temporal EO task models · xBD damage classification | F1: 11.8 -> 50.0 | Helped |
| use TEOChat rather than the specialist model | ABCD zero-shot evaluation · ABCD | score: 95.2 -> 85.6 | Hurt |
| use TEOChat rather than GPT-4o | multi-image VLM comparison · S2Looking detection | F1: 21.5 -> 33.6 | Helped |
| use TEOChat rather than GeoChat | single-image EO evaluation · AID | accuracy: 72.0 -> 80.9 | Helped |
Surprising findings
- Image references in the prompt help even in the bitemporal case, and removing them at test time causes substantial drops.
- Video-LLaVA without fine-tuning on TEOChatlas performs very poorly on temporal tasks (e.g., 16.6% on fMoW RGB), while LLaVA initialization underperforms Video-LLaVA.
Limitations
- Localization underperforms specialist models by a large margin because TEOChat outputs axis-aligned boxes as natural language tokens rather than masks.
- Limited to RGB imagery; does not process multi-spectral bands.
- Sequence length capped at 8 images and inputs resized to 224x224, limiting resolution.
Experimental setup
Datasets: TEOChatlas GeoChat_Instruct fMoW xBD S2Looking QFabric fMoW-Sentinel ABCD CDVQA AID UCMerced LRBEN
Benchmarks: fMoW xBD S2Looking QFabric ABCD CDVQA AID UCMerced LRBEN HRBEN
Metrics: accuracy F1 IoU
Baselines: Video-LLaVA GeoChat SatMAE GPT-4o Gemini 1.5 Pro InternVL2 Qwen2-VL RSVQA RSGPT LHRS-Bot VHM
Builds on: Video-LLaVA LLaVA-1.5 GeoChat
Closest papers by meaning
- EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues (CVPR 2025 · similarity 0.90)
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery (AAAI 2026 · similarity 0.89)
- TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.89)
- VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · similarity 0.88)
- LISAt: Language-Instructed Segmentation Assistant for Satellite Imagery (NeurIPS 2025 · similarity 0.87)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.