Research map / Label-efficient and robust learning / Test-time adaptation
Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MiDl, a test-time adaptation method that makes a pretrained audio-visual model robust to missing modalities on unlabeled egocentric video by combining a mutual-information invariance loss with a self-distillation (KL) loss to the unadapted predictions.
Problem
Multimodal egocentric video models degrade sharply when audio or video modalities are missing at test time, and existing remedies typically require retraining or labeled data.
Approach
Adapt a pretrained multimodal model online and unsupervised at test time: minimize mutual information between predictions and modality-source combinations (invariance) while keeping predictions under full modality close to the original model via KL self-distillation, so performance is preserved when modalities are complete.
Key results
- On Epic-Kitchens with 75% missing, MiDl reaches 46.4% vs 37.9% for the MBT baseline without adaptation.
- With all modalities present (100%), MiDl matches the baseline (29.5% on Epic-Kitchens), preserving full-modality performance.
- Mutual-information loss alone can hurt full-modality performance; the self-distillation term is needed to retain it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MiDl test-time adaptation | MBT · Epic-Sounds, 1-p_AV=25, 50, 75 | accuracy: 45.6 -> 46.8; 37.1 -> 38.8; 28.3 -> 29.8 | Helped |
| Apply MiDl test-time adaptation | MBT · Epic-Kitchens, 1-p_AV=25, 50, 75 | accuracy: 55.5 -> 58.4; 46.8 -> 52.4; 37.9 -> 46.4 | Helped |
| Apply MiDl test-time adaptation | MBT · Epic-Sounds, 1-p_AV=0 | accuracy: 55.1 -> 55.0 | Hurt |
| Apply MiDl test-time adaptation | MBT · Epic-Kitchens, 1-p_AV=0 | accuracy: 63.9 -> 63.7 | Hurt |
| Apply MiDl test-time adaptation | MBT · Epic-Sounds, 1-p_AV=100 | accuracy: 19.5 -> 19.5 | No clear effect |
| Apply MiDl test-time adaptation | MBT · Epic-Kitchens, 1-p_AV=100 | accuracy: 29.5 -> 29.5 | No clear effect |
| Apply KL loss to each modality prediction in addition to audiovisual predictions (ablation) | MiDl · Epic-Sounds | accuracy: 46.8 -> 46.8; 38.8 -> 38.9; 29.8 -> 30.0 | No clear effect |
| Apply KL loss to each modality prediction in addition to audiovisual predictions (ablation) | MiDl · Epic-Kitchens | accuracy: 58.4 -> 57.9; 52.4 -> 52.1; 46.4 -> 46.4 | No clear effect |
Surprising findings
- Using only the mutual-information loss can degrade performance, so the distillation term is essential.
Limitations
- Requires some modal-complete (multimodal) samples at test time; gives no gain when only unimodal streams are present.
- Needs multiple forward passes over modality combinations plus a backward pass, adding test-time compute.
- Requires fixed-size inputs with zero placeholders for missing modalities, tied to the MBT architecture.
Experimental setup
Datasets: Epic-Kitchens Epic-Sounds
Benchmarks: Epic-Kitchens Epic-Sounds
Metrics: accuracy
Baselines: MBT without adaptation unimodal model KL on each modality variant
Builds on: MBT (multimodal bottleneck transformer)
Closest papers by meaning
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.88)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.88)
- X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization (CVPR 2024 · similarity 0.88)
- Object-Shot Enhanced Grounding Network for Egocentric Video (CVPR 2025 · similarity 0.88)
- Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention (NeurIPS 2025 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.