Research map / Architectures and efficiency / Mixture of experts
Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoPE, an asymmetric LoRA with a dedicated poisoning expert trained on hybrid-noise-injected data and masked at inference, to achieve noise robustness for PEFT without data cleaning.
Problem
Parameter-efficient fine-tuning of language models is degraded by noisy training data, and existing remedies rely on costly data cleaning or architectures prone to error propagation.
Approach
Use noise injection (HyNoIse: character-level discrete plus embedding-level continuous noise) to train one B matrix in an asymmetric shared-A multi-B LoRA as a poisoning expert, freeze it, and mask it at inference, with a DyCompEnSate weight compensation for the remaining experts.
Key results
- LoPE (r=8) fine-tuned on noisy data reaches 44.82 MMLU, 14.31 GSM8K and 62.36 ARC-e on LLaMA2-7b, versus 43.08, 11.83 and 56.26 for HydraLoRA (r=4) on the same noisy data.
- On the noise-free Orig setting, LoPE gains only marginally over HydraLoRA (e.g., 46.86 vs 46.10 MMLU at r=4), which the authors attribute to the absence of noise.
- Hybrid HyNoIse outperforms discrete-only, continuous-only and no injection at 3.5%, 5% and 8% noise levels, e.g., 61.86 vs 57.48 for None at 8% noise on PIQA+SIQA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoPE instead of HydraLoRA | LLaMA2-7B · MMLU (Orig) | accuracy (%): 46.10 -> 46.86 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · PIQA (Orig) | accuracy (%): 76.28 -> 76.99 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · SIQA (Orig) | accuracy (%): 52.92 -> 53.58 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · GSM8K (Orig) | accuracy (%): 16.15 -> 17.89 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · ARC-e (Orig) | accuracy (%): 62.61 -> 64.20 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · NSET (Orig) | accuracy (%): 35.99 -> 36.82 | Helped |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · SBH (Orig) | accuracy (%): 54.78 -> 54.46 | Hurt |
| Use LoPE instead of HydraLoRA | LLaMA2-7B · History (Orig) | accuracy (%): 55.95 -> 56.66 | Helped |
| Combine discrete and continuous noise injection instead of injecting no noise (ablation) | LoPE · PIQA and SIQA (average), 5% level | accuracy (%): 59.89 -> 62.66 | Helped |
| Use hybrid noise injection instead of discrete-only injection (ablation) | LoPE · PIQA and SIQA (average), 5% level | accuracy (%): 61.95 -> 62.66 | Helped |
| Use hybrid noise injection instead of continuous-only injection (ablation) | LoPE · PIQA and SIQA (average), 5% level | accuracy (%): 60.71 -> 62.66 | Helped |
| Add HyNoIse to HydraLoRA (ablation) | HydraLoRA · MMLU | accuracy (%): 43.08 -> 40.57 | Hurt |
Surprising findings
- Applying HyNoIse directly to HydraLoRA without the poisoning-expert design lowers performance (e.g., MMLU 40.57 vs 43.08), indicating the architecture and multi-stage training are essential.
- Excessive HyNoIse ratios (10% and above) degrade robustness, giving a non-monotonic relationship between injected noise ratio and performance.
Limitations
- Relies on the LoRA framework, which limits extensibility.
- Discrete noise injection in stage I does not model the distribution of real noisy data.
- Only evaluated with a fixed, modest set of tasks and a single 1-epoch setup; the gains on the clean Orig data are small.
Experimental setup
Datasets: Alpaca-52K
Benchmarks: MMLU (NSET, SBH, History subsets) GSM8K PIQA SIQA ARC-easy
Metrics: accuracy
Baselines: P-Tuning Prefix Tuning AdaLoRA LoRA HydraLoRA
Builds on: LoRA HydraLoRA mixture of experts
Closest papers by meaning
- Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks? (ICML 2025 · similarity 0.88)
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.88)
- HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · similarity 0.88)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.87)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.