Research map / Video / Video-language understanding
Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Animal-Bench, a 13-task, 41,839-QA multiple-choice benchmark for animal-centric video understanding, plus a diffusion-based video editing pipeline that simulates weather and shooting-parameter changes to test robustness of 8 multimodal video models.
Problem
Existing video-language benchmarks are human- or object-centric (animals are about 1% of MVBench), so model ability to understand animal agents, and robustness under realistic filming variations, is unmeasured.
Approach
Builds an animal-centric task system (common tasks plus conservation-relevant special tasks such as predator-prey, social, breeding, and stress/pain detection) with an automated pipeline that filters animal videos from existing datasets and auto-generates multiple-choice QA with designed distractors, and uses diffusion-based outpainting and cropping to simulate snow, frost, distance, and direction
Key results
- VideoChat2 achieves the best average accuracy of 53.66% on Animal-Bench, with 86.75% on object recognition but near-random performance on action localization (21.22%) and action count (29.16%).
- Under simulated weather and shooting-parameter edits, VideoChat2 drops only 3.70% overall while Video-LLaMA drops 8.72%; models are more sensitive to shooting parameters than weather.
- Models show agent bias: VideoChat2's accuracy on animal subsets of MSRVTT-QA/TGIF-QA-derived counting tasks is lower than on the full original datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VideoChat2 rather than the other evaluated models | multimodal video models · Animal-Bench | accuracy (Avg): 53.66 | Helped |
| use VideoChat2 rather than other evaluated models | multimodal video models · Animal-Bench object recognition | accuracy: 86.75 | Helped |
| use VideoChat2 rather than other evaluated models | multimodal video models · Animal-Bench reasoning | accuracy: 68.23 | Helped |
| simulate weather and shooting-parameter variations | VideoChat2 · Animal-Bench robustness evaluation | relative accuracy drop (%): 3.70% | Hurt |
| simulate weather and shooting-parameter variations | Video-LLaMA · Animal-Bench robustness evaluation | relative accuracy drop (%): 8.72% | Hurt |
| reduce input video frames from 16 to 8 (ablation) | VideoChat2 · Animal-Bench action prediction | accuracy: 50.00 -> 46.15↓3.85 | Hurt |
| reduce input video frames from 16 to 8 (ablation) | VideoChat2 · Animal-Bench object count | accuracy: 64.94 -> 69.54↑4.6 | Helped |
| reduce input video frames from 16 to 8 (ablation) | VideoChat2 · Animal-Bench | accuracy (Avg): 53.66 -> 53.64↓0.02 | No clear effect |
| pad non-square video frames before scaling and cropping (ablation) | VideoChat2 · Animal-Bench | accuracy (Avg): 53.66 -> 53.04↓0.62 | Hurt |
| sample distractors from different animal categories (ablation) | VideoChat2 · Animal-Bench object recognition | accuracy: 86.75 -> 97.31 | Helped |
| sample all distractors from the same animal category (ablation) | VideoChat2 · Animal-Bench object recognition | accuracy: 86.75 -> 76.34 | Hurt |
| use more powerful or fine-tuned video encoders instead of frozen CLIP ViT/L-14 (ablation) | multimodal video models · Animal-Bench | average accuracy | Helped |
Surprising findings
- Models answer 'yes' to object existence questions frequently, indicating severe hallucination.
- Several models score below random chance on some animal tasks, and Video-LLaMA is more robustness-sensitive than VideoChat2.
Limitations
- Video editing with Stable Diffusion may introduce artifacts or extra animals that change answers; mitigated by manual selection.
- Robustness tests use only 100 samples per task and the model inference parameters were not varied.
- Data drawn from web and existing datasets, with some tasks having few samples (e.g., 174 to 346).
Experimental setup
Datasets: MammalNet Animal Kingdom LoTE-animal TGIF-QA MSRVTT-QA NExT-QA
Benchmarks: Animal-Bench
Metrics: accuracy relative accuracy drop
Baselines: mPLUG-Owl VideoChat Video-ChatGPT Video-LLaMA Valley Chat-UniVi Video-LLaVA VideoChat2
Builds on: MVBench Stable Diffusion inpainting ChatGPT-based question generation
Closest papers by meaning
- Web-Scale Collection of Video Data for 4D Animal Reconstruction (NeurIPS 2025 · similarity 0.89)
- Meow-Omni 1: A Multimodal Large Language Model for Feline Ethology (NeurIPS 2026 · similarity 0.87)
- MammAlps: A Multi-view Video Behavior Monitoring Dataset of Wild Mammals in the Swiss Alps (CVPR 2025 · similarity 0.87)
- The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition (CVPR 2025 · similarity 0.86)
- AniMo: Species-Aware Model for Text-Driven Animal Motion Generation (CVPR 2025 · similarity 0.86)
- BEAST3D: animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.