Research map / Label-efficient and robust learning / Test-time adaptation
Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies test-time adaptation when a distribution shift affects only one modality of multi-modal data, and proposes modality-specific adapters with a Gumbel-Softmax router that selects which modality to adapt.
Problem
Test-time adaptation of pre-trained multi-modal models when only one modality shifts; existing TTA methods adapt all modalities indiscriminately, causing redundancy and negative transfer.
Approach
A learnable router (Gumbel-Softmax over modality shift logits) blends original and adapted features per modality via lightweight residual adapters, so only the shifted modality is adapted; trained end-to-end with entropy-based pseudo-label self-training and a traverse inference ensemble.
Key results
- On Kinetics50-C with video corruptions under attention-based fusion, the method averages 64.5% accuracy versus 62.5% for READ and 59.9% for the source model.
- On Kinetics50-C with audio corruptions, the method reaches 71.5% average accuracy versus 71.1% for READ under AF.
- Tent and SAR show negative transfer on uni-modal shifts, e.g., Tent drops to 49.4 vs 52.2 source average under late fusion on Kinetics50-C video corruptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Gumbel-softmax routing instead of direct softmax (ablation) | Selective adaptation method · Kinetics50-C video shift: Gaussian noise | accuracy: 52.0 -> 52.6 | Helped |
| Add traverse inference (ablation) | Selective adaptation method · Kinetics50-C video shift: Gaussian noise | accuracy: 51.5 -> 52.6 | Helped |
| Add selective adaptation (ablation) | Selective adaptation method · Kinetics50-C video shift: Gaussian noise | accuracy: 50.3 -> 52.6 | Helped |
| Use selective adapters and routing rather than the source model | CAV-MAE · Kinetics50-C video shift: rain | accuracy: 67.5 -> 68.7 | Helped |
Surprising findings
- Tent and SAR degrade below the unadapted source model on uni-modal shifts (negative transfer).
- Larger parameter count (1.180M vs 0.226M for Tent) does not compromise efficiency; Tent runs out of memory on VGGSound-C due to gradient graph storage.
Limitations
- Evaluated only on two audio-visual datasets with synthetic corruptions
- Batch size sensitivity noted: accuracy declines as batch size grows from 8 to 128
- Improvement on VGGSound-C with video shift is small
Experimental setup
Datasets: Kinetics50 VGGSound Kinetics50-C VGGSound-C
Benchmarks: Kinetics50-C VGGSound-C
Metrics: accuracy error rate parameter count time cost per epoch
Baselines: Tent ETA SAR MM-TTA READ
Builds on: Tent ETA SAR READ MM-TTA Gumbel-Softmax
Closest papers by meaning
- Attention Bootstrapping for Multi-Modal Test-Time Adaptation (AAAI 2025 · similarity 0.90)
- Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises (ICLR 2025 · similarity 0.88)
- Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.87)
- Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.86)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.