Research map / Vision-language and multimodal / Multimodal LLMs
Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgroMind, a 13-task, 4-dimension benchmark of 28,482 QA pairs over 20,850 agricultural remote sensing images, and evaluates 24 LMMs plus human and random baselines.
Problem
Existing benchmarks for large multimodal models in agricultural remote sensing have limited scene diversity and oversimplified task design, so LMM capabilities in agriculture are poorly measured.
Approach
Builds a multi-source agricultural RS VQA benchmark by integrating nine public datasets and one private parcel dataset, organized into a hierarchy of spatial perception, object understanding, scene understanding, and scene reasoning, with rule-based and human-written questions and strict-match scoring (semantic similarity for open-ended items).
Key results
- GPT-4o achieves the highest overall accuracy at 43.14%, about 10 points above the human average of 33.15%.
- 17 of 24 evaluated models underperform humans overall, per the paper's main-results claim.
- Most LMMs score poorly on bounding-box/spatial tasks (BD) and counting; e.g., GPT-4o scores 23.75% on CO versus 78.55% on OC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of human participants | AgroMind evaluation · AgroMind | overall accuracy: 33.15 -> 43.14; 9.99% | Helped |
| Scale LLaVA-NeXT from 8B to 13B | LLaVA-NeXT · AgroMind | overall accuracy: 31.53 -> 30.69; 0.84% | Hurt |
| Use a 34B LLaVA-NeXT model rather than a 26B InternVL2 model | LLaVA-NeXT and InternVL2 · AgroMind | overall accuracy: 37.25 vs. 35.52 | Hurt |
| Use LMMs rather than humans for visual prompt reasoning | AgroMind evaluation · AgroMind VPR | accuracy: 12.33 vs. 33.05 | Hurt |
| Use LMMs rather than humans for counting | AgroMind evaluation · AgroMind CO | accuracy: 28.88 vs. 23.75 | Hurt |
| Use GPT-4o rather than human participants for boundary detection | AgroMind evaluation · AgroMind BD | accuracy: 40.83 vs. 33.52 | Hurt |
| Increase model parameter size | LMMs on AgroMind · AgroMind | overall accuracy: 31.53 -> 30.69 | Mixed |
Surprising findings
- Human participants (mostly students) score below several LMMs and GPT-4o, with humans outperforming models on some scene-level and visual reasoning tasks.
- Larger models do not reliably outperform smaller ones: LLaVA-NeXT-34B underperforms InternVL2-26B, and LLaVA-NeXT-13B is below its 8B variant on overall accuracy.
Limitations
- Human baseline uses roughly twenty student volunteers who may lack specialized agricultural knowledge, making the human comparison weak.
- Multi-image questions are converted to single images by concatenation, which may disadvantage multi-image models.
- Open-ended answers are scored via semantic similarity, which adds evaluation uncertainty; the text does not detail the judge.
Experimental setup
Datasets: OAM-TCD OilPalmUAV Agriculture Vision Challenge PhenoBench Maize Tassel Identification (Iflytek AI Competition 2024) IP102 ACFR Orchard Fruit Dataset 2018 AI Challenge Dataset CropHarvest Global Parcel Dataset (private)
Benchmarks: AgroMind
Metrics: accuracy overall average accuracy across 13 tasks
Baselines: Random Human GeoChat GeoLLaVA-8K Mantis InstructBLIP-Vicuna-7B
Builds on: MMMU UrBench
Closest papers by meaning
- AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark (NeurIPS 2025 · similarity 0.90)
- AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning (NeurIPS 2026 · similarity 0.90)
- Fields of The World: A Machine Learning Benchmark Dataset for Global Agricultural Field Boundary Segmentation (AAAI 2025 · similarity 0.89)
- HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing (AAAI 2024 · similarity 0.87)
- Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types (AAAI 2024 · similarity 0.87)
- GTPBD: A Fine-Grained Global Terraced Parcel and Boundary Dataset (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.