Research map / Vision-language and multimodal / Multimodal LLMs
AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgMMU, a real-world agricultural VLM benchmark (746 MCQs and 746 OEQs) distilled from expert-grower dialogues, plus AgBase, a 57k-fact fine-tuning corpus, showing large gaps for VLMs and gains from LoRA fine-tuning LLaVA-1.5-7B.
Problem
Existing multimodal benchmarks do not test knowledge-intensive agricultural understanding from real user queries, and VLMs lack domain-specific agricultural training data.
Approach
Builds QA pairs from authentic grower-to-USDA-Cooperative-Extension-expert conversations via an LLM-based categorization, knowledge extraction, and QA generation pipeline with human verification, offering both MCQ and open-ended formats plus a disjoint large fine-tuning set.
Key results
- Gemini 1.5 Pro and GPT-o4-mini lead; GPT-o4-mini averages 37.0 on OEQs and 86.5 on MCQs, while the best open model LLaMA-3.2-11B reaches 24.9 OEQ average.
- LoRA fine-tuning LLaVA-1.5-7B on AgBase improves open-ended question performance by up to 11.6% on average.
- Open-sourced models trail proprietary ones by more than 10 to 20% on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune LLaVA-1.5-7B on AgBase | LLaVA-1.5-7B · AgMMU-OEQs | average performance: up to 11.6% | Helped |
| use 10,000 LLaVA instruction-tuning samples with AgBase instead of 57,079 LLaVA SFT samples (ablation) | LLaVA-1.5-7B · AgMMU | overall accuracy: 0.24 -> 0.25 | Helped |
| add full original user-query context to species-identification training samples (ablation) | LLaVA-1.5-7B Species SFT · AgMMU species identification | accuracy | Hurt |
| increase the number of MCQ answer choices from four to six (ablation) | GPT-4o · AgMMU MCQ subset | accuracy: 5-10% | Hurt |
Surprising findings
- Fine-tuning on a smaller 10k mix gave slightly higher accuracy (0.25) than the 57k mix (0.24), suggesting a well-curated smaller mixture may be more effective.
- Accuracy drops by 5-10% from 4 to 6 MCQ options, suggesting process-of-elimination effects.
Limitations
- Benchmark covers only 746 MCQ and 746 OEQ items and a limited set of VLMs.
- Simple fine-tuning yields smaller gains on multi-statement management and symptom questions.
- Knowledge extraction and QA generation rely on GPT-4o and Llama-70B, which may introduce errors.
Experimental setup
Datasets: AgBase LLaVA-Instruct-150K Extension.org expert-user conversations (116,231 dialogues)
Benchmarks: AgMMU-MCQs AgMMU-OEQs
Metrics: accuracy correct/total harmonic mean F-score (SimpleQA-style)
Baselines: GPT-o4-mini Gemini 1.5 Pro Claude 3 Haiku LLaVA-1.5 LLaVA-NeXT LLaVA-OneVision Cambrian-1 InternVL2 Qwen-VL VILA1.5 LLaMA-3.2-11B
Builds on: SimpleQA MMMU LLaVA
Closest papers by meaning
- AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning (NeurIPS 2026 · similarity 0.94)
- Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind (NeurIPS 2025 · similarity 0.90)
- MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversations (NeurIPS 2025 · similarity 0.90)
- Empowering and Assessing the Utility of Large Language Models in Crop Science (NeurIPS 2024 · similarity 0.88)
- Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding (CVPR 2024 · similarity 0.87)
- BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.