Research map / Label-efficient and robust learning / Test-time adaptation
Attention Bootstrapping for Multi-Modal Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ABPEM, a multi-modal test-time adaptation method that bootstraps cross-attention toward self-attention statistics and uses principal (top-k class) entropy minimization, evaluated on Kinetics50-C and VGGSound-C.
Problem
Test-time distribution shift in multi-modal models widens the gap between intra-modality and inter-modality attention, hindering modality fusion; entropy minimization gradients are also noisy without labels.
Approach
Model raw self- and cross-attention scores as Gaussians and minimize KL divergence so cross-attention distributions align with self-attention anchors (attention bootstrapping), and compute entropy only over the top-k most likely classes per sample (principal entropy minimization) to reduce gradient noise.
Key results
- On Kinetics50-C corrupted video (severity 5), ABPEM averages 65.0% vs 62.5% for READ and 59.9% for the raw model.
- On VGGSound-C corrupted audio, ABPEM averages 36.1% vs 32.4% for READ and 25.6% for raw.
- Ablation shows attention bootstrapping gives the largest accuracy gain over the base tunable-layer model, with principal entropy minimization adding a further boost over vanilla entropy minimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ABPEM instead of READ | CAV-MAE test-time adaptation · Kinetics50-C, corrupted video | accuracy: 62.5 -> 65.0 | Helped |
| Use ABPEM instead of READ | CAV-MAE test-time adaptation · Kinetics50-C, corrupted audio | accuracy: 71.1 -> 72.5 | Helped |
| Use ABPEM instead of READ | CAV-MAE test-time adaptation · VGGSound-C, corrupted video | accuracy: 56.9 -> 57.5 | Helped |
| Use ABPEM instead of READ | CAV-MAE test-time adaptation · VGGSound-C, corrupted audio | accuracy: 32.4 -> 36.1 | Helped |
| Add attention bootstrapping (ablation) | ABPEM test-time adaptation · Kinetics50-C | accuracy | Helped |
| Replace vanilla entropy minimization with principal entropy minimization (ablation) | ABPEM test-time adaptation · Kinetics50-C | accuracy | Helped |
| Add vanilla entropy minimization instead of the basic self-supervised objective (ablation) | ABPEM test-time adaptation · Kinetics50-C | accuracy | Helped |
| Vary principal entropy class count k from 6 to 10 (ablation) | ABPEM test-time adaptation · Kinetics50-C | accuracy: 70.64, 70.87, 70.96, 70.95, 70.94 | No clear effect |
Surprising findings
- MMT and Tent substantially degrade accuracy below the unadapted model on several VGGSound-C corruptions (e.g., Tent falls to 4.5% average on corrupted audio vs 25.6% raw).
Limitations
- Evaluated only on two-modality audio-video benchmarks with synthetic corruptions.
- Requires hyperparameter k chosen per dataset (about 8 for Kinetics50-C, 30 for VGGSound-C).
Experimental setup
Datasets: Kinetics50-C VGGSound-C Kinetics VGGSound
Benchmarks: Kinetics50-C VGGSound-C
Metrics: prediction accuracy (%) samples per second number of tunable parameters
Baselines: Raw Tent MMT EATA SAR READ
Builds on: Tent CAV-MAE READ EATA SAR
Closest papers by meaning
- Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data (ICML 2025 · similarity 0.90)
- Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.88)
- Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises (ICLR 2025 · similarity 0.85)
- Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition (NeurIPS 2026 · similarity 0.85)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.85)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.