Research map / Label-efficient and robust learning / Test-time adaptation
Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AEO, an entropy-aware test-time adaptation framework (adaptive entropy weighting plus cross-modal prediction discrepancy) for multimodal open-set TTA, evaluated on a new benchmark built from EPIC-Kitchens, HAC, Kinetics-100-C and nuScenes.
Problem
Test-time adaptation of a multimodal source model to a target domain that contains unknown classes absent from the source label space, where existing unimodal open-set TTA methods and entropy minimization degrade unknown-class detection.
Approach
Observes that the entropy gap between unknown and known samples correlates with open-set detection performance, and uses an adaptive tanh-weighted entropy loss that minimizes entropy for likely-known samples and maximizes it for likely-unknown ones, plus a cross-modality prediction discrepancy term that is maximized for unknown and minimized for known samples.
Key results
- On EPIC-Kitchens (video+audio), AEO improves the Source mean H-score from 34.11 to 56.18 (+22.07), vs. READ at 37.66 and UniEnt at 44.85.
- On HAC averaged over four modality combinations, AEO raises H-score by 13.70, 12.12, 9.12 and 10.82 points over Source.
- On nuScenes Day-to-Night, AEO achieves H-score 55.51 vs. Source 53.76 and MM-TTA 50.99; FPR95 is reduced from 47.97 to 44.90.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AEO instead of the Source model | Multimodal open-set TTA on EPIC-Kitchens · EPIC-Kitchens mean (D1/D2/D3 shifts) | H-score: 34.11 -> 56.18 | Helped |
| Use AEO instead of READ | Multimodal open-set TTA on nuScenes · nuScenes Day→Night | H-score: 53.67 -> 55.51 | Helped |
| Add UAE to the method without UAE or AMP (ablation) | AEO on HAC · HAC mean, video+audio | H-score: 34.61 -> 47.79 | Helped |
| Add AMP to the method without UAE or AMP (ablation) | AEO on HAC · HAC mean, video+audio | H-score: 34.61 -> 45.49 | Helped |
| Add UAE to AEO with AMP only (ablation) | AEO on HAC · HAC mean, video+audio | H-score: 45.49 -> 48.31 | Helped |
| Add AMP to AEO with UAE only (ablation) | AEO on EPIC-Kitchens · EPIC-Kitchens mean | H-score: 55.01 -> 56.18 | Helped |
| Use AEO instead of Tent | Multimodal open-set TTA on EPIC-Kitchens · EPIC-Kitchens mean (D1/D2/D3 shifts) | H-score: 16.52 -> 56.18 | Helped |
| Use AEO instead of Source | Multimodal open-set TTA under corruption · Kinetics-100-C mean, severity 5 | H-score: 37.66 -> 60.07 | Helped |
Surprising findings
- Tent, a standard entropy-minimization TTA method, degrades open-set performance well below the unadapted source model (e.g., mean H-score 16.52 vs. 34.11 on EPIC-Kitchens).
- OSTTA, a unimodal open-set TTA method, underperforms the source model on several EPIC-Kitchens splits.
Limitations
- Benchmark is constructed from existing datasets by designating specific classes or domains as unknown, rather than from naturally collected open-set data.
- Hyperparameters alpha, beta, gamma1, gamma2 are set by hand and sensitivity is only explored for W_ada parameters.
Experimental setup
Datasets: EPIC-Kitchens HAC Kinetics-600 Kinetics-100-C nuScenes
Benchmarks: EPIC-Kitchens HAC Kinetics-100-C nuScenes
Metrics: Acc IoU FPR95 AUROC H-score
Baselines: Source Tent SAR OSTTA UniEnt READ MM-TTA
Builds on: Tent SAR UniEnt OSTTA READ
Closest papers by meaning
- Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises (ICLR 2025 · similarity 0.89)
- Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM (CVPR 2025 · similarity 0.89)
- ReservoirTTA: Prolonged Test-time Adaptation for Evolving and Recurring Domains (NeurIPS 2025 · similarity 0.89)
- Modality-Collaborative Test-Time Adaptation for Action Recognition (CVPR 2024 · similarity 0.88)
- Attention Bootstrapping for Multi-Modal Test-Time Adaptation (AAAI 2025 · similarity 0.88)
- PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.