Research map / Label-efficient and robust learning / Test-time adaptation
Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuMi, a test-time adaptation method for multimodal wild TTA that smooths adaptation with IQR-based sample selection, selects low-entropy samples using unimodal entropy, and adds a mutual information sharing loss, evaluated on corrupted Kinetics50 and VGGSound.
Problem
Test-time adaptation of multimodal models under complex noise: simultaneous corruptions across modalities, missing modalities, and mixtures of weak and strong distribution shifts in the target stream, where existing TTA methods collapse.
Approach
Gradually expand the set of adapted samples using an IQR-based criterion whose width grows with iteration (smoothing from weak to strong OOD), filter low-entropy samples using both multimodal and unimodal entropy to keep samples with rich multimodal information, and align per-modality predictions with the complement distribution via a KL-based mutual information sharing loss.
Key results
- On Kinetics50-C (severity 5, video corruption), SuMi reaches 63.9 average accuracy versus 63.5 for the next best method READ and 60.2 for the source model.
- On VGGSound-C strong OOD scenarios, SuMi reaches 19.7 average accuracy versus 17.4 for EATA and 15.6 for the source model.
- Ablation on Kinetics50-C at severity 5 with 50% strong OOD: the full model reaches 52.0 versus 31.7 for the source model, with IQR smoothing giving the largest single gain (45.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SuMi instead of EATA | CAV-MAE · Kinetics50-C video corruption severity 5, Avg. | accuracy: 61.9 -> 63.9 | Helped |
| apply SuMi instead of READ | CAV-MAE · Kinetics50-C audio corruption severity 5, Avg. | accuracy: 71.6 -> 71.9 | Helped |
| apply SuMi instead of EATA | CAV-MAE · Kinetics50-C strong OOD severity 5, Avg. | accuracy: 30.3 -> 33.4 | Helped |
| apply SuMi instead of READ | CAV-MAE · VGGSound-C video corruption severity 5, Avg. | accuracy: 56.4 -> 57.3 | Helped |
| apply SuMi instead of EATA | CAV-MAE · VGGSound-C audio corruption severity 5, Avg. | accuracy: 32.5 -> 33.2 | Helped |
| apply SuMi instead of EATA | CAV-MAE · VGGSound-C strong OOD severity 5, Avg. | accuracy: 17.4 -> 19.7 | Helped |
| add IQR smoothing (ablation) | SuMi · Kinetics50-C with 50% strong OOD, severity 3 | accuracy: 37.1 -> 52.1 | Helped |
| add unimodal assistance (ablation) | SuMi · Kinetics50-C with 50% strong OOD, severity 3 | accuracy: 37.1 -> 49.4 | Helped |
| add mutual information sharing (ablation) | SuMi · Kinetics50-C with 50% strong OOD, severity 3 | accuracy: 37.1 -> 47.4 | Helped |
| add IQR smoothing (ablation) | SuMi · VGGSound-C with 50% strong OOD, severity 3 | accuracy: 25.7 -> 33.8 | Helped |
| add unimodal assistance (ablation) | SuMi · VGGSound-C with 50% strong OOD, severity 3 | accuracy: 25.7 -> 31.1 | Helped |
| add mutual information sharing (ablation) | SuMi · VGGSound-C with 50% strong OOD, severity 3 | accuracy: 25.7 -> 29.8 | Helped |
Surprising findings
- Most existing TTA methods (Tent, SAR, SoTTA, DeYO, CEMA) perform worse than the source model on strong OOD samples, and several collapse to single-digit accuracy on VGGSound-C.
- For unimodal entropy, lower entropy does not necessarily correspond to better accuracy; samples in a mid entropy band yielded better results for audio and video.
Limitations
- Mutual information sharing loss is applied only in the first t0 iterations for strong OOD adaptation, a heuristic choice.
- Evaluated mainly on two corrupted benchmarks built from Kinetics50 and VGGSound with synthetic corruptions; hyperparameters are set per dataset.
Experimental setup
Datasets: Kinetics50 VGGSound CMU-MOSI CH-SIMS UPMC-Food101
Benchmarks: Kinetics50-C VGGSound-C CMU-MOSI UPMC-Food101 CH-SIMS
Metrics: accuracy F1
Baselines: Tent EATA SAR SoTTA DeYO CEMA READ
Builds on: Tent EATA SAR READ CAV-MAE
Closest papers by meaning
- Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.89)
- Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data (ICML 2025 · similarity 0.88)
- Unified Entropy Optimization for Open-Set Test-Time Adaptation (CVPR 2024 · similarity 0.88)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.88)
- SNAP: Low-Latency Test-Time Adaptation with Sparse Updates (NeurIPS 2025 · similarity 0.88)
- Bridging the Gap for Test-Time Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.