Research map / Vision-language and multimodal / Multimodal LLMs
MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MIRAGE, a multimodal agricultural benchmark built from real AskExtension expert-user consultations (single-turn identification/management and multi-turn clarify-or-respond), evaluates 20+ VLMs with an LLM-judge ensemble, and LoRA-fine-tunes Qwen2.5-VL.
Problem
Existing VLM benchmarks use well-specified inputs, closed-set taxonomies, and short-form answers, so they do not capture underspecified, image-grounded expert consultations in knowledge-intensive domains like agriculture.
Approach
Derive a benchmark from real user-expert agricultural conversations with expert-authored long-form answers, covering single-turn multimodal identification/management (with contextual location/time subset) and multi-turn clarify-or-respond decisions, scored by an ensemble of reasoning LLM judges.
Key results
- GPT-4.1 achieves the highest identification accuracy of 43.9% on MMST-ID, while the best open model Qwen2.5-VL-72B reaches 29.83%.
- LoRA fine-tuning Qwen2.5-VL-3B raises seen-entity ID accuracy from 22.3% to 28.4%, but unseen-entity accuracy stays around 14.6%.
- On MMMT zero-shot, GPT-4o reaches 62.98% decision accuracy, and CoT raises it to 65.50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply chain-of-thought prompting | GPT-4o · MIRAGE-MMMT | Decision accuracy (%): 62.98 -> 65.50; 2.52 | Helped |
| Apply chain-of-thought prompting | Claude 3.7 Sonnet · MIRAGE-MMMT | Decision accuracy (%): 57.80 -> 62.40; 4.60 | Helped |
| Apply chain-of-thought prompting | LLaMA 4 Maverick 17B · MIRAGE-MMMT | Decision accuracy (%): 53.75 -> 59.80; 6.05 | Helped |
| Apply chain-of-thought prompting | LLaMA 4 Scout 17B · MIRAGE-MMMT | Decision accuracy (%): 49.81 -> 56.00; 6.19 | Helped |
| Apply chain-of-thought prompting | LLaMA 3.2 90B · MIRAGE-MMMT | Decision accuracy (%): 45.09 -> 50.40; 5.31 | Helped |
| Apply chain-of-thought prompting | Qwen 72B · MIRAGE-MMMT | Decision accuracy (%): 31.33 -> 37.40; 6.07 | Helped |
| Fine-tune with LoRA on MIRAGE-MMST training data | Qwen2.5-VL-3B · MIRAGE-MMST ID seen entities | Identification accuracy (%): 22.3\% ... 28.4\% | Helped |
| Fine-tune with LoRA on MIRAGE-MMST training data | Qwen2.5-VL-3B · MIRAGE-MMST ID unseen entities | Identification accuracy (%) | No clear effect |
| Include metadata such as location and time | GPT-4.1 · MIRAGE-MMST Contextual | Identification accuracy (%): +1.6\% | Helped |
| Include metadata such as location and time | Qwen2.5-VL-72B · MIRAGE-MMST Contextual | Identification accuracy (%): +0.6\% | Helped |
| Add user-stated goal to classifier inputs | Logistic regression classifier · MIRAGE-MMMT | Decision accuracy (%): 69.79\% → 71.34\% | Helped |
| Add oracle goal state to classifier inputs | Logistic regression classifier · MIRAGE-MMMT | Decision accuracy (%): 89.27\% | Helped |
Surprising findings
- Metadata (location and time) yields minimal identification gains and sometimes slightly degrades performance.
- A rule-based oracle with full goal-state access reaches 98.45% decision accuracy, showing the difficulty lies in inferring missing context.
Limitations
- Benchmark reflects mainly smallholder and backyard gardening scenarios from AskExtension rather than commercial agriculture.
- Does not simulate interactive dialogue with real users or user simulators; user follow-up turns are text-only.
- Generalization to unseen entities remains poor even after fine-tuning.
Experimental setup
Datasets: AskExtension expert-user conversations (MIRAGE-MMST training split) MIRAGE-MMMT dialogues
Benchmarks: MIRAGE-MMST Standard MIRAGE-MMST Contextual MIRAGE-MMMT
Metrics: Identification Accuracy Reasoning Score Accuracy Relevance Completeness Parsimony Decision Accuracy Clarify Relevance Respond Relevance Fleiss kappa
Baselines: GPT-4.1 GPT-4o Claude 3.7 Sonnet Qwen2.5-VL-72B InternVL3-78B Gemma-3-27B LLaMA 4 Scout
Builds on: LLM-as-a-Judge MMMU AgMMU CROP
Closest papers by meaning
- AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark (NeurIPS 2025 · similarity 0.90)
- AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning (NeurIPS 2026 · similarity 0.88)
- Empowering and Assessing the Utility of Large Language Models in Crop Science (NeurIPS 2024 · similarity 0.86)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.84)
- VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images (NeurIPS 2024 · similarity 0.84)
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.