Research map / Adversarial robustness and security / Certified and empirical robustness

Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach

ICML 2025 · Adversarial robustness and security › Certified and empirical robustness · approach: Adversarial training recipes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that MoE experts are more adversarially vulnerable than the router, proposes RT-ER (KL-regularized robust training of the second-top expert), a dual-model linear combination of standard and robust MoE with certified bounds, and a bi-level joint training (JTDMoE) that improves SA and RA on CIFAR-10 and TinyImageNet.

Problem

Adversarial robustness of Mixture of Experts models is poor, and existing robust training methods for MoE either ignore component-level vulnerability or sacrifice substantial standard accuracy.

Approach

Identify experts (not the router) as the vulnerable MoE component, robustify only the second-top expert via an added KL term, and combine a standard and a robust MoE with a smoothing parameter alpha, jointly trained with a bi-level scheme to recover clean accuracy while keeping certified robustness bounds.

Key results

• On CIFAR-10 (ResNet18 experts), RT-ER reaches 69.09% PGD robust accuracy vs 53.74% for AT, at 77.81% SA.
• JTDMoE on CIFAR-10 at alpha=0.7 achieves 92.29% SA and 74.62% PGD RA, versus 89.81% SA and 67.81% RA for the pre-trained dual-model.
• Under AutoAttack on CIFAR-10, JTDMoE improves RA from 33.64% to 54.55% over the pre-trained dual-model.

Measured effects reported in the paper

Surprising findings

• Experts are far more vulnerable than the router: RA-E drops below 4% on CIFAR-10 under PGD while RA-R stays above 50%.
• Setting the KL weight beta=9 collapses all metrics to 10% (random guessing) when training from scratch.

Limitations

• Theoretical bounds rely on an assumption of Lipschitz continuity of experts and router that holds only under stated scenarios.
• Large SA loss for single-model robust training (e.g., RT-ER SA 77.81% vs ST 92.14%) motivates the dual-model, which is only partly recovered.
• Dual-model is built by combining pretrained models for efficiency rather than training from scratch; experiments limited to CIFAR-10 and TinyImageNet at small scale.

Experimental setup

Datasets: CIFAR-10 TinyImageNet ImageNet

Benchmarks: CIFAR-10 TinyImageNet ImageNet

Metrics: standard accuracy robust accuracy RA-E RA-R certified robustness bound

Baselines: Standard training (ST) Adversarial training (AT) TRADES AdvMoE Pre-trained Dual-Model

Builds on: PGD adversarial training TRADES AdvMoE Lipschitz-based certified robustness

Closest papers by meaning

• On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.88)
• $\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts (ICML 2024 · similarity 0.88)
• On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.88)
• Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.88)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.88)
• MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.88)