Research map / Adversarial robustness and security / Attacks on text and graphs

Breaking Adversarial Transferability in Fine-Tuned Speech Recognition

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: Adversarial prompts for language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TransferBreaker, an adversarial fine-tuning framework (Base-AFT, LJ-Regularization, HybridGrad-AFT) that suppresses transferability of base-model adversarial audio perturbations to fine-tuned ASR models.

Problem

Adversarial perturbations crafted on a public pretrained ASR base model transfer strongly to privately fine-tuned black-box target models, degrading WER severely; existing adversarial fine-tuning is over-constraining and fails under this transfer threat and adaptive surrogate attacks.

Approach

Restrict adversarial fine-tuning to perturbations generated on the frozen base model (Base-AFT), anchor a latent-invariance regularizer to the frozen base representation (LJ-Regularization), and for adaptive attacks mix base and target gradients in PGD (HybridGrad-AFT).

Key results

• Average adversarial WER on base-model attacks drops from 92.6 (fine-tuned) to 27.8 with TransferBreaker, versus 42.8 for Standard-AFT.
• Whisper-medium on Arabic: Base-Adv WER reduced from 110.0 (FT) to 36.5 (TransferBreaker), with clean WER 26.2.
• Under surrogate-based adaptive attacks, Polish Whisper-medium Surrogate-Adv WER falls from 57.6 (Base-AFT) to 44.1 (TransferBreaker).

Surprising findings

• Standard adversarial fine-tuning remains substantially less effective than Base-AFT against base-model transferred attacks, suggesting over-constraining for ASR.
• Adversarial transferability from base to fine-tuned models persists even when fine-tuning data scales to about 1,000 hours, with adversarial WER decreasing only 4-5 points.

Limitations

• HybridGrad-AFT introduces a trade-off on base-model attacks, only partially recovered by LJR.
• Evaluation limited to three languages, short fine-tuning data (~30 hours), and untargeted SNR-constrained attacks.
• Adaptive attacker assumed to use a surrogate trained with the same framework on its own data; other adaptive strategies not fully covered.

Experimental setup

Datasets: Mozilla Common Voice 24 Multilingual LibriSpeech (MLS) ParlaSpeech (Polish)

Benchmarks: Common Voice 24 test split (Polish, Portuguese, Arabic)

Metrics: WER CER

Baselines: Standard-AFT FT (fine-tuned without defense) Base-AFT Base+Std-AFT

Builds on: adversarial fine-tuning projected gradient descent momentum iterative attack

Closest papers by meaning

• Enhancing Adversarial Transferability with Adversarial Weight Tuning (AAAI 2025 · similarity 0.86)
• Towards Transferable Adversarial Attacks with Centralized Perturbation (AAAI 2024 · similarity 0.85)
• Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment (NeurIPS 2025 · similarity 0.85)
• Initialization Matters for Adversarial Transfer Learning (CVPR 2024 · similarity 0.85)
• LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate (AAAI 2024 · similarity 0.85)
• Speech Robust Bench: A Robustness Benchmark For Speech Recognition (ICLR 2025 · similarity 0.85)