Research map / Vision-language and multimodal / Multimodal LLMs
RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RealBirdID, a bird species identification benchmark where models must either identify the species or abstain with a reason (e.g., vocalization needed, low quality, view obstructed), and evaluates encoders and MLLMs on classification and abstention.
Problem
Fine-grained bird species identification from a single image is often unanswerable, yet existing evaluations only test answerable cases, so models are not measured on principled abstention with reasons.
Approach
A validation benchmark pairing expert-vetted unanswerable iNaturalist bird observations (with labeled abstention reasons) with answerable Research Grade images, plus new metrics: abstention tradeoff (UA/A), information-gain classification performance (IG), and an accuracy-vs-abstention calibration AUC, applied to both CLIP-style encoders and MLLMs.
Key results
- Species accuracy on the answerable set is at most 17.0% for encoders (BioCLIP) and at most 12.7% for MLLMs (Gemini-2.5 Pro), with GPT-5 at 10.4%.
- BioCLIP achieves the highest IG (68.9) but SigLIP has the best UA/A (53.2), showing classification ability does not imply abstention ability.
- MLLMs rarely abstain by default; Qwen2.5-VL abstention rate varies widely under instruction rewording, and models almost never predict vocalization as the abstention reason.
Surprising findings
- Virtually no model predicts vocalization as the abstention reason, even when it is the ground-truth issue.
- Using range maps degrades MLLM abstention tradeoff even though it improves classification.
Limitations
- Unanswerable subset is heavily long-tailed across genera, with 85% of UA images from 61 genera.
- Labeling of unanswerable examples depends on expert judgment and may be disputed.
- Multi-image observations where the clear bird appears only in later frames are not handled.
Experimental setup
Datasets: iNaturalist SINR range maps RealBirdID
Benchmarks: RealBirdID
Metrics: IG (information gain accuracy-vs-information curve AUC) UA/A (abstention tradeoff AUC) AUC and accuracy at species and genus levels abstention rate Pearson correlation
Baselines: CLIP MetaCLIP WildCLIP SigLIP BioCLIP Qwen2.5-VL Llama-3.2 Vision InternVL3 Gemma-3 Gemini-2.5 Pro GPT-5 PaliGemma
Builds on: TreeGT DARTS nlg2choice SINR
Closest papers by meaning
- BirdCollect: A Comprehensive Benchmark for Analyzing Dense Bird Flock Attributes (AAAI 2024 · similarity 0.88)
- Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition (AAAI 2025 · similarity 0.86)
- CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing (NeurIPS 2025 · similarity 0.85)
- WildlifeMapper: Aerial Image Analysis for Multi-Species Detection and Identification (CVPR 2024 · similarity 0.84)
- Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis (CVPR 2025 · similarity 0.84)
- Generating the Wild: Individual-Consistent Image-to-Video Generation for Wildlife (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.