Research map / Vision-language and multimodal / Visual question answering
Language-aware Visual Semantic Distillation for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoDistill, a language-aware (goal-driven) VideoQA framework using a Language-Aware Gate (LA-Gate) in place of cross-attention for differentiable sparse frame sampling and vision refinement, pretrained on ~11M video-caption pairs.
Problem
Goal-free video perception and cross-attention based language-vision fusion in VideoQA miss question-relevant frames and multi-scale visual cues, and encourage language shortcut solutions and bias.
Approach
A language-aware gate that scales visual tokens by their cosine similarity to the question embedding (no direct text injection into visual features), used both for Gumbel-softmax top-K frame selection and for multi-level vision refinement; answers are generated from visual embeddings only.
Key results
- On EgoTaskQA normal split, VideoDistill (trained from scratch) reaches 45.02% overall accuracy vs 42.20% for HCRN (prior SOTA), a gain of about 2.82 points.
- On AGQA v2 test, VideoDistill reaches 55.80% overall vs 54.39% for MIST-CLIP, a 1.41-point gain.
- Replacing LA-Gate with cross-attention lowers EgoTaskQA accuracy from 45.02% to 40.94% and MSRVTT-QA from 44.20% to 41.17% in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoDistill instead of MIST-CLIP | VideoQA models · AGQA v2 test set (All) | accuracy: 54.39 -> 55.80 | Helped |
| Use VideoDistill instead of HCRN | VideoQA models · EgoTaskQA normal split (all) | accuracy: 42.20 -> 45.02 | Helped |
| Remove sparse sampling and vision refinement, using evenly sampled 16 frames and a 3-layer transformer (ablation) | VideoDistill · EgoTaskQA | accuracy: 29.10 -> 45.02 | Hurt |
| Remove sparse sampling and vision refinement, using evenly sampled 16 frames and a 3-layer transformer (ablation) | VideoDistill · MSRVTT-QA | accuracy: 23.92 -> 44.20 | Hurt |
| Remove differentiable sparse sampling and use evenly sampled 16 frames (ablation) | VideoDistill · EgoTaskQA | accuracy: 41.70 -> 45.02 | Hurt |
| Remove differentiable sparse sampling and use evenly sampled 16 frames (ablation) | VideoDistill · MSRVTT-QA | accuracy: 40.92 -> 44.20 | Hurt |
| Replace hard top-k sampling with weighted sums of dense frames (ablation) | VideoDistill · EgoTaskQA | accuracy: 38.95 -> 45.02 | Hurt |
| Replace hard top-k sampling with weighted sums of dense frames (ablation) | VideoDistill · MSRVTT-QA | accuracy: 38.16 -> 44.20 | Hurt |
| Remove vision refinement and use a 3-layer transformer to fuse vision and language (ablation) | VideoDistill · EgoTaskQA | accuracy: 33.44 -> 45.02 | Hurt |
| Remove vision refinement and use a 3-layer transformer to fuse vision and language (ablation) | VideoDistill · MSRVTT-QA | accuracy: 26.51 -> 44.20 | Hurt |
| Replace LA-Gates with cross-attention layers (ablation) | VideoDistill · EgoTaskQA | accuracy: 40.94 -> 45.02 | Hurt |
| Replace LA-Gates with cross-attention layers (ablation) | VideoDistill · MSRVTT-QA | accuracy: 41.17 -> 44.20 | Hurt |
Surprising findings
- The language-only variant of VideoDistill drops sharply on EgoTaskQA (e.g., object accuracy goes to 0), indicating reduced reliance on language shortcuts.
Limitations
- Relies on frozen CLIP visual encoder and large-scale pretraining data
- Action-category performance on AGQA is limited, attributed to language correlations
Experimental setup
Datasets: WebVid10M HD-VILA EgoCLIP MSRVTT-QA MSVD-QA EgoMCQ MSRVTT multiple-choice EgoTaskQA AGQA v2
Benchmarks: MSRVTT-QA MSVD-QA EgoMCQ MSRVTT multiple-choice EgoTaskQA AGQA v2 text-to-video retrieval
Metrics: accuracy
Baselines: HCRN PMT AIO MIST-CLIP ClipBERT HGA HME PSAC VisualBERT BERT
Builds on: CLIP TimeSformer ClipBERT Gumbel-Softmax
Closest papers by meaning
- Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.91)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.90)
- Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames (NeurIPS 2025 · similarity 0.90)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.90)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.90)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.