Research map / Vision-language and multimodal / Multimodal LLMs

LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LithoBench, a 10,000-instance multi-level (five cognitive levels, MCQ and open-ended) benchmark built from GF-2 lithology imagery with an expert-in-the-loop semi-automated pipeline, and evaluates ~14 LVLMs plus LoRA fine-tuned Qwen3.5-9B and InternVL3.5-8B.

Problem

Existing remote sensing and geoscience VLM benchmarks do not evaluate knowledge-intensive lithology interpretation across multiple levels of geological reasoning, from visual identification to mechanism explanation and practical application.

Approach

A lithology-specific benchmark where structured expert-guided image descriptions, dual-modal retrieval (text knowledge base via BGE and image knowledge base via CLIP for realistic distractors), hierarchical Bloom-based QA generation, and LLM-as-a-judge plus expert quota-constrained filtering yield 10,000 balanced instances.

Key results

• LoRA-fine-tuned InternVL3.5-8B-HF achieves the best MCQ-all accuracy among listed models at 0.9785 and OPEN_J 1.9740 vs 1.6512 for GPT-5.4 (zero-shot best).
• Fine-tuned Qwen3.5-9B reaches MCQ_all 0.9825, outperforming all unfine-tuned models including GPT-5.4 (0.9300).
• Best open-ended BLEU-4 reaches only 0.108 and ROUGE-L 0.257 even after fine-tuning, far below scores on UCM-Captions and RSICD.

Surprising findings

• Open-ended automatic metrics (ROUGE-L, BLEU-4) do not always track LLM-judge scores; some models with reasonable lexical overlap score lower on judge evaluation.
• Cross-model open-ended answers are highly similar (cosine 0.83-0.93), suggesting shared descriptive patterns rather than fine-grained reasoning.

Limitations

• Gaofen-2 imagery from a single region in northwestern China with 12 coarse categories, limiting geographic and taxonomic generality.
• Heavy reliance on GPT-5.4 and LLM judges for description and QA generation and filtering may introduce model-specific bias.

Experimental setup

Datasets: GF-2 imagery (112,163 patches, 12 lithology categories) LithoBench training data

Benchmarks: LithoBench

Metrics: accuracy BLEU-4 ROUGE-L OPEN_w/oJ OPEN_J JudgeOverallScore

Baselines: Claude-Sonnet-4.6 Gemini-3-Pro GPT-5.4 Grok-4 Qwen3.5-27B Qwen3.5-397B-A17B Intern-S1-Pro InternVL3.5-241B-A28B Kimi-K2.5 GLM-4.6V

Builds on: LLM-as-a-Judge Bloom's taxonomy BGE-Large-En-v1.5 CLIP

Closest papers by meaning

• PEACE: Empowering Geologic Map Holistic Understanding with MLLMs (CVPR 2025 · similarity 0.88)
• GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing (CVPR 2026 · similarity 0.86)
• TerraScope: Pixel-Grounded Visual Reasoning for Earth Observation (CVPR 2026 · similarity 0.85)
• SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts (AAAI 2026 · similarity 0.84)
• Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models (ICLR 2026 · similarity 0.84)
• Learnable Earth Parser: Discovering 3D Prototypes in Aerial Scans (CVPR 2024 · similarity 0.83)