Research map / Video / Video segmentation and tracking
Video Event Extraction with Multi-View Interaction Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MID, a knowledge-distillation framework (self-RKD for inter-object relations in the SlowFast video encoder and LKD for layer-wise cross-modal supervision in the decoder) that improves video event extraction on VidSitu without extra parameters.
Problem
Video event extraction (verb classification and semantic role/argument generation) lacks explicit learning strategies for modeling inter-object interactions and inter-modality (text-video) alignment.
Approach
Self-relational KD transfers inter-object interaction matrices (measured by MMD, dot product, or Gaussian RBF) from deeper to shallow SlowFast layers, and layer-to-layer KD distills per-layer cross-attention outputs from a teacher decoder into a randomly initialized student decoder as extra cross-modal supervision.
Key results
- On VidSitu verb classification, MID (MMD) reaches 53.11 Acc@1 and 47.98 F1@5 on test, versus 52.39 and 44.93 for the strongest OSE variant.
- On semantic role prediction, MID (MMD) achieves 49.78 CIDEr averaged over 10 seeds, compared with 48.46 for OSE-pixel/disp + OME.
- The paper reports 2.94% F1@5 gain on verb classification and 1.32% CIDEr gain on role prediction, with no additional parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-RKD with naive MMD | MID · VidSitu test | F1@5: 44.93 -> 47.98 | Helped |
| Add LKD (ablation) | MID (MMD) · VidSitu validation | CIDEr: 49.13 -> 49.78 | Helped |
| Add self-RKD with naive MMD (ablation) | MID · VidSitu validation | CIDEr: 48.60 -> 49.78 | Helped |
| Use Gaussian RBF for self-RKD instead of naive MMD (ablation) | MID · VidSitu validation | CIDEr: 49.78 ± 1.91 -> 49.12 ± 1.87 | Hurt |
| Use dot product for self-RKD instead of naive MMD (ablation) | MID · VidSitu validation | CIDEr: 49.78 ± 1.91 -> 48.71 ± 2.03 | Hurt |
| Remove LKD (ablation) | MID (MMD) · VidSitu validation | ROUGE-L: 42.56 -> 42.11 | Hurt |
Surprising findings
- Dot-product self-RKD performs worse than the other kernels and sometimes below the baseline on verb classification, which the authors attribute to poor relational modeling.
Limitations
- Relies on an external object tracker (VidVRD) to provide bounding boxes, which the method depends on.
- Gains are modest (around 1-3 points) and role prediction results have high variance across seeds.
- Video-LLaMA was evaluated only zero-shot due to limited compute.
Experimental setup
Datasets: VidSitu
Benchmarks: VidSitu
Metrics: Accuracy@1 Accuracy@5 Recall@5 F1@5 CIDEr CIDEr-Verb CIDEr-Arg ROUGE-L
Baselines: TimeSformer I3D SlowFast OSE-pixel + OME OSE-pixel/disp + OME OSE-pixel/disp + OME + OIE GPT2 Video-LLaMA
Builds on: SlowFast OSE (object state embedding) VidSitu Relational knowledge distillation VidVRD
Closest papers by meaning
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.86)
- Cross-modal Multi-task Learning for Multimedia Event Extraction (AAAI 2025 · similarity 0.86)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.85)
- Streaming Detection of Queried Event Start (NeurIPS 2024 · similarity 0.84)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.84)
- Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.