Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Mixture of Adversarial LoRAs: Boosting Robust Generalization in Meta-Tuning

NeurIPS 2024 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Normalization and unsupervised meta-learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMT, adversarial meta-tuning of a pool of LoRAs with input and singular-value/vector perturbations plus test-time merging, to improve clean and adversarial out-of-domain few-shot classification of ViTs.

Problem

Robust generalization of pre-trained vision transformers in out-of-domain few-shot image classification, where models must handle both distribution shifts and adversarial attacks from a single source domain.

Approach

Meta-tune a pool of LoRAs in parallel at different adversarial budgets, applying adversarial perturbations to inputs and to the singular values and vectors (via LoRA initialized from SVD with SAM-style perturbation), then merge LoRAs per task at test time using intra-class compactness and inter-class divergence scores with singular value trimming.

Key results

• On Meta-Dataset 5-way 1-shot, AMT reaches 65.16 average clean accuracy vs 60.35 for PMF-style PM baseline, with up to 12.92% gain on Omniglot in the tuning-free setting.
• Under PGD-10 adversarial evaluation (5-way 1-shot), AMT improves average robust accuracy from 13.54 (PM) to 28.09, with up to 49.72% gain on Omniglot.
• AMT-FT with test-time fine-tuning improves over PMF and ATTNSCALE by 3.92% and 4.3% respectively in clean accuracy on Meta-Dataset.

Measured effects reported in the paper

Surprising findings

• Naively perturbing with adversarial attacks and weight-level SAM in the original LoRA space performs worse than perturbing singular values and vectors, and removing the singular-value perturbation hurts OOD generalization.
• Adversarially robust pre-trained ViTs (AdvPre) gain in in-domain clean accuracy after AMT, outperforming clean pre-trained models.

Limitations

• Adversarial budget epsilon for each LoRA module must be manually set.
• Only l-infinity bounded perturbations are explored.
• Single source domain (ImageNet) setting only.

Experimental setup

Datasets: ImageNet Meta-Dataset BSCD-FSL fine-grained CD-FSL datasets (CUB, Cars, Places, Plantae)

Benchmarks: Meta-Dataset BSCD-FSL CD-FSL fine-grained datasets

Metrics: clean few-shot accuracy adversarial robust accuracy (PGD-10) AutoAttack robust accuracy

Baselines: PMF ATTNSCALE StyleAdv FLUTE ETT NFTS SAM ALT RandConv

Builds on: PMF LoRA StyleAdv SAM PGD adversarial training

Closest papers by meaning

• Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors (NeurIPS 2022 · similarity 0.88)
• f-Domain Adversarial Learning: Theory and Algorithms (ICML 2021 · similarity 0.88)
• RobustGenBench: A Benchmark for Robust Generalization to Adversarial and Common Perturbations, with Applications to Vision and Vision-Enabled Large Language Models (NeurIPS 2026 · similarity 0.88)
• Kernel Learning with Adversarial Features: Numerical Efficiency and Adaptive Regularization (NeurIPS 2025 · similarity 0.87)
• Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.87)
• Explaining the role of Intrinsic Dimensionality in Adversarial Training (ICML 2025 · similarity 0.86)