Research map / Vision-language and multimodal / Visual question answering
VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VLM4Bio, a ~469K question-answer benchmark over 30K fish, bird, and butterfly images, and evaluates 12 pretrained VLMs zero-shot on five organismal-biology tasks plus prompting and hallucination tests.
Problem
Whether pretrained vision-language models hold the scientific knowledge needed to answer biologically relevant trait-discovery questions from organism images without fine-tuning.
Approach
A multi-task VQA benchmark for organismal biology (species classification, trait identification, trait grounding, trait referring, trait counting) built from museum fish, CUB bird, and Heliconius butterfly images, with expert-annotated bounding boxes, plus analyses of answer-choice difficulty, prompting strategies, and reasoning-hallucination tests (FCT and NOTA).
Key results
- Best open-ended species classification accuracy is very low (e.g., 17.40% for GPT-4V on Bird-10K), while MC accuracy reaches 82.58% (GPT-4V, Bird-10K).
- Trait grounding and referring accuracies are near chance for most models (e.g., Bird-500 grounding 8.1% for GPT-4V), indicating weak visual localization.
- BioCLIP outperforms all large VLMs on Bird-10K MC (93.93%) and Butterfly-10K MC (62.32%), while the largest VLMs remain near chance on hard butterfly set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multiple-choice rather than open-ended species questions | Pretrained VLMs · Species classification across Fish-10K, Bird-10K, and Butterfly-10K | accuracy | Helped |
| Use easier answer choices from different genera rather than harder fine-grained choices | Pretrained VLMs · Species classification, Fish, Bird, and Butterfly difficulty sets | accuracy | Helped |
| Use biologically fine-tuned BioCLIP rather than general pretrained VLMs | Species classification models · Bird-10K and Butterfly-10K | accuracy | Helped |
| Fine-tune with biological images and scientific names rather than use OpenAI CLIP weights | CLIP · Species classification, Bird-10K and Butterfly-10K | accuracy | Helped |
| Add zero-shot chain-of-thought prompting (ablation) | GPT-4o · Fish-Prompting species classification | accuracy: 79.00 -> 86.00 | Helped |
| Add zero-shot chain-of-thought prompting (ablation) | GPT-4o · Butterfly-Prompting species classification | accuracy: 56.40 -> 74.60 | Helped |
| Add contextual prompting (ablation) | GPT-4o · Bird-Prompting species classification | accuracy: 97.60 -> 98.60 | Helped |
| Add dense-caption prompting (ablation) | GPT-4V · Butterfly-Prompting species classification | accuracy: 13.20 -> 49.60 | Helped |
| Add dense-caption prompting (ablation) | LLaVA-v1.5-7B · Fish-Prompting species classification | accuracy: 41.60 -> 26.00 | Hurt |
| Increase answer-choice difficulty from easy to medium or hard (ablation) | Pretrained VLMs · Fish, Bird, and Butterfly species classification | accuracy | Hurt |
| Increase image resolution (ablation) | Pretrained VLMs · Fish-10K species classification, MC | accuracy | Helped |
| Increase image resolution (ablation) | Pretrained VLMs · Bird-10K species classification, MC | accuracy | No clear effect |
Surprising findings
- Instruct-Vicuna-13B reaches 89.98% on bird trait identification, higher than GPT-4V, yet trait identification is inflated by always-present traits like eye and head.
- Chain-of-thought and dense-caption prompting help GPT-4V/GPT-4o but not smaller models, and GPT-4V often refuses to answer open-ended species questions.
Limitations
- Covers only three organism groups (fish, birds, butterflies) due to resource constraints.
- Some label noise remains in the dataset and was not manually verified across all images.
- Proprietary models such as Gemini and Claude were not evaluated.
Experimental setup
Datasets: Fish-10K (FishAIR) Bird-10K (CUB-200-2011) Butterfly-10K (Jiggins Heliconius Collection) iNatLoc500 Phenoscape
Benchmarks: VLM4Bio
Metrics: micro-averaged accuracy agreement score
Baselines: GPT-4V LLaVA-v1.5-7B LLaVA-v1.5-13B CogVLM MiniGPT-4 BLIP-2 FLAN-T5-XL InstructBLIP BioCLIP CLIP
Builds on: BioCLIP CUB-200-2011 GroundingDINO Segment Anything Model
Closest papers by meaning
- BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · similarity 0.88)
- HoneyBee: Data Recipes for Vision-Language Reasoners (CVPR 2026 · similarity 0.88)
- BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025 · similarity 0.87)
- CoralVQA: A Large-Scale Visual Question Answering Dataset for Coral Reef Image Understanding (NeurIPS 2025 · similarity 0.86)
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.