Research map / Label-efficient and robust learning / Knowledge distillation
Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RADI, a ranking distillation framework for open-ended video QA that uses a teacher trained on incomplete labels to produce answer rankings, with adaptive soft-margin pairwise and partial listwise distillation losses.
Problem
Open-ended video QA benchmarks provide typically one labeled answer per question, so multi-label answer sets are insufficiently labeled and existing methods treat unlabeled correct answers as negatives.
Approach
Use a teacher trained with incomplete labels to generate answer rankings (rather than probability distributions or hard pseudo labels) and distill them via parameter-free pairwise soft-margin (Sinkhorn-based uncertainty-adaptive margins from MC dropout) and partial listwise (hot/cold list ranking-based sampling) losses.
Key results
- RADI-P improves over prior SOTA on iVQA by 3.9% Acc@1 (43.5 vs 39.6 for FrozenBiLM) and RADI-L by 3.3%, with no extra trainable parameters.
- On MSRVTT-QA, RADI-P reaches 48.2 Acc@1 vs 47.0 for FrozenBiLM and 48.3 for All-in-one; on TGIF-FrameQA RADI-L reaches 70.0 vs 68.6.
- Under single-labeled-answer training on iVQA, RADI-P achieves 43.2 Acc@1 and 68.3 Hit@5, outperforming Vanilla KD (41.3) and DKD (41.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive pairwise ranking distillation | FrozenBiLM · iVQA | Acc@1: 39.6 -> 43.5 | Helped |
| Add partial listwise ranking distillation | FrozenBiLM · iVQA | Acc@1: 39.6 -> 42.9 | Helped |
| Add adaptive pairwise ranking distillation | FrozenBiLM · ActivityNet-QA | Acc@1: 43.2 -> 44.1 | Helped |
| Add partial listwise ranking distillation | FrozenBiLM · MSVD-QA | Acc@1: 54.8 -> 56.0 | Helped |
| Add adaptive pairwise ranking distillation | FrozenBiLM · MSRVTT-QA | Acc@1: 47.0 -> 48.2 | Helped |
| Add partial listwise ranking distillation | FrozenBiLM · TGIF-FrameQA | Acc@1: 68.6 -> 70.0 | Helped |
| Add uncertainty estimation to pairwise margin weighting (ablation) | RADI pairwise · iVQA | Acc@1: 41.9 -> 41.5 | Helped |
| Add OT-based margin adaptation to uncertainty weighting (ablation) | RADI pairwise · iVQA | Acc@1: 41.5 -> 43.5 | Helped |
| Replace full-list distillation with random partial-list sampling (ablation) | RADI listwise · iVQA | Acc@1: 41.3 -> 41.3 | No clear effect |
| Use exponential ranking-based sampling instead of full-list distillation (ablation) | RADI listwise · iVQA | Acc@1: 41.3 -> 42.3 | Helped |
| Use Zipf ranking-based sampling instead of full-list distillation (ablation) | RADI listwise · iVQA | Acc@1: 41.3 -> 42.9 | Helped |
| Initialize the RADI-P student from scratch rather than individual initialization (ablation) | RADI-P · iVQA | Acc@1: 43.5 -> 42.9 | Hurt |
Surprising findings
- Continual training with only the classification loss leads to performance decline due to overfitting, while adding only the ranking loss yields further gains.
- Label smoothing generally reduces Acc@1, and pseudo labeling degrades quickly as more pseudo labels are added, while FitNet and RKD feature/relation distillation hurt performance.
Limitations
- Pairwise margin adaptation requires computing pairwise uncertainty matrices, truncated to top 2500 answers for efficiency.
- Performance depends on teacher quality, though the method is more stable than KD under a weaker teacher.
Experimental setup
Datasets: iVQA ActivityNet-QA MSVD-QA MSRVTT-QA TGIF-FrameQA
Benchmarks: iVQA ActivityNet-QA MSVD-QA MSRVTT-QA TGIF-FrameQA
Metrics: Acc@1 Hit@5 nDCG@5
Baselines: FrozenBiLM All-in-one VIOLET MERLOT Co-Token SViTT Just Ask SiaSamRea Vanilla KD FitNet RKD DKD
Builds on: FrozenBiLM ListNet Knowledge distillation Sinkhorn algorithm LambdaLoss
Closest papers by meaning
- Knowledge-Constrained Answer Generation for Open-Ended Video Question Answering (AAAI 2023 · similarity 0.88)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.86)
- STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering (AAAI 2024 · similarity 0.86)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (CVPR 2025 · similarity 0.85)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.85)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.