Research map / Vision-language and multimodal / Cross-modal learning

$\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time

NeurIPS 2025 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AVROBUSTBENCH, a test-time robustness benchmark with 75 co-occurring, correlated bimodal audio-visual corruptions across four datasets, evaluates supervised and self-supervised AV models and online TTA methods, and proposes a entropy-penalized cross-modal TTA baseline.

Problem

Robustness of audio-visual recognition models to distribution shifts occurring simultaneously in audio and visual modalities at test time is underexplored; existing benchmarks address single modalities.

Approach

Apply 15 correlated corruption types (5 severities each) simultaneously to audio and video of AudioSet, VGGSound, Kinetics-Sounds and Epic-Kitchens test sets, and propose a TTA method that updates joint-encoder QKV attention with entropy minimization on reliable, diverse low-entropy samples to enable on-the-fly cross-modal fusion.

Key results

• On VGGSound at severity 5, CAV-MAE's mean accuracy falls from 65.50% clean to 35.54% under corruptions; the proposed TTA reaches 44.06% mean accuracy, outperforming READ (38.74%) and SuMi (35.56%).
• On Kinetics-Sounds, the proposed TTA reaches 62.33% mean accuracy versus READ at 62.30% and source at 58.15%, with a 88.10% clean source accuracy.
• Self-supervised models show low robustness, e.g., Wav2CLIP mean accuracy 4.99% on VGGSound at severity 5 with relative robustness 0.21.

Measured effects reported in the paper

Surprising findings

• Existing TTA methods such as TENT and SAR often degrade below the source model on VGGSound under bimodal corruptions, attributed to LayerNorm overfitting.
• Audio-visual models are less robust on Digital corruptions, and AudioSet models show milder drops than VGGSound/Kinetics-Sounds models, attributed to pre-training data noise.

Limitations

• Not all models evaluated on all datasets (EquiAV not on AudioSet; EquiAV checkpoints re-trained by the authors; Kinetics-Sounds models trained from scratch).
• TTA gains are limited on Kinetics-Sounds and the gap to clean source accuracy remains wide.
• Corruptions are synthetic approximations of real-world shifts.

Experimental setup

Datasets: AudioSet VGGSound Kinetics-Sounds Epic-Kitchens Freesound MUSIC AVSBench-S4

Benchmarks: AudioSet VGGSound Kinetics-Sounds Epic-Kitchens AVSBench-S4 MUSIC

Metrics: accuracy mAP absolute robustness relative robustness mIoU F-score SDR SIR SAR

Baselines: TENT SAR RPL EATA READ SuMi Source (CAV-MAE)

Builds on: ImageNet-C TENT EATA READ CAV-MAE

Closest papers by meaning

• Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation (ICLR 2025 · similarity 0.89)
• VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.88)
• Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives (ICLR 2025 · similarity 0.88)
• AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs (ICML 2026 · similarity 0.88)
• Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
• Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.87)