Research map / Video / Video-language understanding
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens Continente, Larisa Markeeva, Dylan Sunil Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova and 12 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune SeViLA on Perception Test train split | SeViLA · Perception Test mc-vQA val | top-1 accuracy: 46.2 -> 62.0 | Improved |
| 8-shot prompting instead of 0-shot | Flamingo-3B · Perception Test mc-vQA val | top-1 accuracy: 43.6 -> 45.8 | Improved |
| scale Flamingo from 3B to 80B | Flamingo · Perception Test mc-vQA val 0-shot | top-1 accuracy: 43.6 -> 41.6 | Hurt |
| use video-language models instead of simple frequency baseline | Flamingo / SeViLA 0-shot or 8-shot · Perception Test mc-vQA val | top-1 accuracy: 45.8 vs 51.0 (8-shot frequency) | Hurt |
| add audio features to video features (ablation) | ActionFormer · Perception Test action localisation | mAP: 14.74 -> 15.56 | Improved |
| add audio features to video features (ablation) | ActionFormer · Perception Test sound localisation | mAP: 13.04 -> 15.46 | Improved |
| use simple SiamFC tracker instead of more complex Stark | object tracking · Perception Test | average IoU: 0.56 -> 0.67 | Improved |
| use TAP-Net instead of static baseline | point tracking · Perception Test | average Jaccard: 0.361 -> 0.401 | Improved |
| moving camera instead of static camera | SiamFC · Perception Test object tracking | average IoU: 0.70 / 0.69 -> 0.42 / 0.54 (static dummy / SiamFC) | Hurt |
Closest papers by meaning
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.91)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.91)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.91)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.90)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.90)
- PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
