Research map / Video / Video-language understanding
Unbiasing through Textual Descriptions: Mitigating Representation Bias in Video Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UTD, a pipeline that uses VLM-generated frame descriptions and LLM concept extraction to measure representation bias (object, temporal, common-sense vs dataset) in 12 video benchmarks, releases object-debiased test splits, and benchmarks 30 video models on them.
Problem
Video benchmarks may be dominated by representation biases (object/scene or single-frame cues), so high scores may reflect object recognition shortcuts rather than true video understanding.
Approach
Use frame-wise textual descriptions from a VLM, filtered into objects, verbs, and activities by an LLM, as an intermediate representation to measure concept, temporal, and common-sense/dataset biases, then remove object-biased test samples that text-embedding-based classifiers solve correctly across all nine model variants.
Key results
- Object-only descriptions reach 63.3% on UCF101 and 45.9% on Kinetics-400 under common-sense bias, showing substantial object representation bias.
- Debiased UTD-splits reduce accuracy of video models notably (e.g., videomae-B-K400 on K400 drops from 64.3 to 45.7 accuracy on the UTD split), with VideoMAE most robust and All-in-one most affected.
- Object-biased samples removed range from 6% (LSMDC-like low) to 73% (UCF101) of test splits; nine-model agreement Fleiss' kappa is 77.8-93.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate video models on the UTD-split instead of the full test set | VideoMAEv2-B-K710-fnK710 · UCF101 | accuracy: 99.0 -> 98.1 | Hurt |
| Evaluate video models on the UTD-split instead of the full test set | VideoMAE-H-UH · SomethingSomethingv2 | accuracy: 67.4 -> 65.5 | Hurt |
| Evaluate video models on the UTD-split instead of the full test set | UMT-L-K710-fnK710 · Kinetics-400 | accuracy: 89.0 -> 78.7 | Hurt |
| Evaluate video-language models on the UTD-split instead of the full test set | UMT-L-17M · MSRVTT | Recall@1: 43.6 -> 31.1 | Hurt |
| Evaluate video-language models on the UTD-split instead of the full test set | UMT-L-17M · DiDeMo | Recall@1: 46.3 -> 35.6 | Hurt |
| Evaluate video-language models on the UTD-split instead of the full test set | UMT-L-17M · ActivityNet | Recall@1: 45.9 -> 38.7 | Hurt |
| Use max-score-frame rather than sequence-of-frames representations (ablation) | UTD common-sense bias evaluation · Video datasets in Table 3 | performance: SSv2 7.4 vs 6.4; Kinetics400 48.0 vs 48.0; ActivityNet 21.5 vs 26.6 | Mixed |
Surprising findings
- Larger backbones and models trained on more data are more robust to object bias in action classification, but this trend does not hold for text-to-video retrieval, where they show higher object bias.
- SSv2 shows almost no performance drop on debiased splits, and verbs-only classification drops much less on SSv2 and MiT than on other datasets.
Limitations
- VLM hallucination rate of about 6% in object descriptions from the user study on UCF frames.
- Bias estimates depend on the chosen proxy models (text embedding classifiers, linear probes), which approximate the maximum over all models.
- Debiasing only removes object-biased samples; it does not address scene or temporal bias in the retained data.
Experimental setup
Datasets: UCF101 Something-Something v2 Kinetics-400 Kinetics-600 Kinetics-700 Moments In Time MSRVTT DiDeMo ActivityNet LSMDC YouCook2 Spoken Moments In Time
Benchmarks: UCF101 Something-Something v2 Kinetics-400 Kinetics-600 Kinetics-700 Moments In Time MSRVTT DiDeMo ActivityNet LSMDC YouCook2 Spoken Moments In Time
Metrics: accuracy recall@1 Fleiss' kappa
Baselines: VideoMAE VideoMAE2 All-in-one Unmasked Teacher VideoMamba InternVid
Builds on: VideoGLUE RESOUND
Closest papers by meaning
- ConViS-Bench: Estimating Video Similarity Through Semantic Concepts (NeurIPS 2025 · similarity 0.87)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.87)
- Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers (CVPR 2025 · similarity 0.87)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.86)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.86)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.