Research map / Vision-language and multimodal / Multimodal LLMs
EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EarthDial, a multi-sensor, multi-resolution, multi-temporal remote-sensing VLM built on InternVL with a Phi-3-mini LLM, trained on an 11M-pair EO instruction dataset.
Problem
Generic VLMs and existing geospatial VLMs handle RGB at fixed resolution and few sensor modalities, and lack multi-spectral, SAR, multi-temporal support for Earth observation tasks.
Approach
A unified EO conversational VLM combining adaptive high-resolution tiling, a band-wise data fusion module for multi-spectral/SAR inputs, and temporal token stacking, trained in three stages on an 11M-pair instruction dataset generated largely from labels with InternLM-XComposer2.
Key results
- On xBD temporal image classification, EarthDial reaches 96.37% accuracy versus 67.95% for GPT-4o and 53.32% for GeoChat.
- On BigEarthNet multi-spectral classification, EarthDial scores 69.94 vs 49 for GPT-4o; on SoSAT-LCZ42 it scores 60.72 vs 15.53.
- On RSVQA-LRBEN, EarthDial averages 92.70 vs 90.70 for GeoChat and 89.19 for LHRS-Bot; on Methane plume classification it reaches 77.09% vs 40.93% for GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-stage pre-training (ablation) | EarthDial · GeoChat-Instruct | mAP@0.5 (multiple referred objects): +5% | Helped |
| Add adaptive high resolution after multi-stage pre-training (ablation) | EarthDial · GeoChat-Instruct | mAP@0.5 (multiple objects): 11.91 -> 15.03 | Helped |
| Use bilinear interpolation instead of average pooling for spectral fusion (ablation) | EarthDial · BigEarthNet (MS) and TreeSatAI | accuracy: 47.66 -> 67.01; 49.09 -> 56.93 | Helped |
| Use bilinear interpolation instead of max-pooling for spectral fusion (ablation) | EarthDial · BigEarthNet (MS) | accuracy: 34.62 -> 67.01 | Helped |
| Use full fine-tuning instead of LoRA rank 128 (ablation) | EarthDial · Urban Tree Crown Detection (ZS) | mAP@0.5 (multiple objects): 2.6 -> 6.7 | Helped |
| Use full fine-tuning instead of LoRA rank 128 (ablation) | EarthDial · Swimming Pool Dataset (ZS) | mAP@0.5 (large objects): 9.45 -> 24.90 | Helped |
| Train EarthDial for multispectral inputs rather than using RGB inputs | EarthDial · BigEarthNet | classification accuracy: +1.75% | Helped |
| Use EarthDial instead of GPT-4o | EarthDial · STARCOP methane plume presence classification | accuracy: 40.93% -> 77.09%; 32.16% | Helped |
| Use EarthDial instead of GPT-4o | EarthDial · UHI-AD (Landsat8) | accuracy: 22.68% -> 56.77% | Helped |
| Use EarthDial instead of GPT-4o | EarthDial · QuakeSet | accuracy: 55.86 -> 57.53 | Helped |
Surprising findings
- Fine-tuning InternVL2 on GeoChat-Instruct was needed for a fair rotated-box comparison, indicating generic VLMs lack rotated bounding box output.
- On several zero-shot detection and captioning benchmarks, EarthDial trails GeoChat or InternVL2 on individual columns (e.g., UCAS-AOD region captioning, Swimming Pool small objects).
Limitations
- Struggles with ambiguous scenes, subtle change detection, and similar-looking temporal classes (e.g., office vs multi-unit residential).
- Limited SAR ship training data leads to weaker ship detection performance.
- Several results are reported with GPT-4o and GeoChat as the only comparators, and some baselines lack multi-spectral support.
Experimental setup
Datasets: SkyScript SatlasPretrain GeoChat-Instruct AID UCMerced WHU-RS19 BigEarthNet xBD fMoW SoSAT-LCZ42 TreeSatAI SAR-Ship
Benchmarks: AID UCMerced WHU-19 BigEarthNet xBD fMoW SoSAT-LCZ42 TreeSatAI SAR-Ship GeoChat-Instruct NWPU VHR-10 Swimming Pool
Metrics: accuracy mAP@0.5 mAP@0.25 ROUGE-1 ROUGE-L METEOR recall
Baselines: GPT-4o GeoChat InternVL2-8B InternVL2-4B LHRS-Bot EarthGPT Qwen-VL MiniGPTv2
Builds on: InternVL InternVL 1.5 AnyRes LLaVA
Closest papers by meaning
- TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data (ICLR 2025 · similarity 0.90)
- GeoChat: Grounded Large Vision-Language Model for Remote Sensing (CVPR 2024 · similarity 0.89)
- Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery (AAAI 2026 · similarity 0.89)
- GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolution (NeurIPS 2025 · similarity 0.89)
- Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents (ICLR 2026 · similarity 0.88)
- TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.