Research map / Adversarial robustness and security / Attacks on text and graphs
Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies adversarial vulnerability of audio-visual classifiers on Kinetics-Sounds, proposes temporal invariance and modality misalignment attacks plus an efficient audio-visual adversarial training framework with curriculum scheduling.
Problem
Audio-visual models are vulnerable to adversarial attacks, and existing attacks adapted from single-modality methods ignore temporal consistency and cross-modal correlation; adversarial training for multi-modal video is also computationally expensive.
Approach
Attacks that exploit temporal redundancy (variance-based feature-consistency regularizer with input transformations) and cross-modal misalignment (minimizing audio-visual feature cosine similarity), plus adversarial training using shared universal perturbations over sampled frame segments and a cyclic data- and model-level curriculum.
Key results
- TIA exceeds PAM by 15.7% and TIA+MMA (TMA) reaches 95.2% average black-box attack success across eight audio-visual models on Kinetics-Sounds.
- Ensemble TMA attacks achieve 100% attack success rate on all eight victim models.
- The proposed adversarial training improves robustness by 2.28% on average over the runner-up CRMT-AT under TMA white-box attacks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine TIA and MMA into TMA | audio-visual adversarial attacks · Kinetics-Sounds, black-box transfer across 8 models | attack success rate: 95.2% | Helped |
| Use TIA instead of PAM | audio-visual adversarial attacks · Kinetics-Sounds, single-model transfer | attack success rate: +15.7% | Helped |
| Add MMA to TIA to form TMA | audio-visual adversarial attacks · MIT-MUSIC, black-box transfer | attack success rate: TIA: 95.2%; MMA: 93.7%; TMA: 97.1% | Helped |
| Use TMA instead of the runner-up attack | audio-visual adversarial attacks · MIT-MUSIC, black-box transfer | attack success rate: 97.1% vs 85.1% | Helped |
| Use the proposed defense tricks with adversarial training | audio-visual adversarial training · Kinetics-Sounds | successful defense rate: +2.28% on average versus CRMT-AT | Helped |
| Increase sampling ratio from 5% to 40% (ablation) | TMA adversarial training · Kinetics-Sounds | successful defense rate: 72.1 -> 83.0 | Helped |
| Increase sampling ratio from 5% to 40% (ablation) | TMA adversarial training · Kinetics-Sounds | training time: 6.2 -> 34.2 | Hurt |
| Increase sampling ratio beyond 30% (ablation) | TMA adversarial training · Kinetics-Sounds | successful defense rate: 82.9 -> 83.0 -> 83.0 | No clear effect |
| Add a synchronized 20% data-and-model masking schedule (ablation) | adversarial curriculum training on RsV · Kinetics-Sounds | successful defense rate: +3.8% | Helped |
| Use a linear curriculum scheduler instead of no scheduler (ablation) | adversarial curriculum training on RsV · Kinetics-Sounds | successful defense rate: +0.8% | Helped |
| Use a cosine curriculum scheduler instead of no scheduler (ablation) | adversarial curriculum training on RsV · Kinetics-Sounds | successful defense rate: +1.4% | Helped |
| Increase the number of I-FGSM attack iterations (ablation) | I-FGSM · Kinetics-Sounds, black-box transfer | attack success rate | Hurt |
Surprising findings
- Momentum (MI-FGSM) does not consistently improve transferability for audio-visual models; it decreases FGSM performance by 2.3% for sum-fusion models.
- Adversarial examples transfer less between cosine-aligned audio and visual features: higher transferability correlates with lower audio-visual cosine similarity.
Limitations
- Experiments limited to a small set of simple audio-visual classifiers (VGG/AlexNet/ResNet) on Kinetics-Sounds and MIT-MUSIC.
- Demo on VideoLLaMA2 is limited to 100 examples and a qualitative prompt-based evaluation.
Experimental setup
Datasets: Kinetics-Sounds MIT-MUSIC
Benchmarks: Kinetics-Sounds MIT-MUSIC
Metrics: attack success rate classification success rate (defense) cosine similarity training time
Baselines: FGSM I-FGSM MI-FGSM MIG PAM AT DCFL SSDF Mixup CRMT-AT
Builds on: FGSM I-FGSM MI-FGSM PAM PGD adversarial training
Closest papers by meaning
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time (NeurIPS 2025 · similarity 0.88)
- FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models (NeurIPS 2026 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
- AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models (ICLR 2025 · similarity 0.85)
- Mutual-Modality Adversarial Attack with Semantic Perturbation (AAAI 2024 · similarity 0.85)
- R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.