Research map / Large language models / Language, safety and interpretability
Multiple Choice Learning of Low-Rank Adapters for Language Modeling
Victor Letzelter, Hugo Malard, Mathieu Fontaine, Gaël Richard, Slim Essid, Andrei Bursuc, Patrick Perez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multiple LoRA hypotheses trained with WTA (LoRA-MCL, K=7) instead of single LoRA-MLE (r=8) | Qwen2-Audio · AudioCaps test loss | test loss (NLL): 2.203 -> 1.932 | Improved |
| add multiple LoRA hypotheses trained with WTA (LoRA-MCL, K=7) instead of single LoRA-MLE (r=8) | Qwen2-Audio · Clotho test loss | test loss (NLL): 2.812 -> 2.612 | Improved |
| increase number of hypotheses K from 3 to 7 (ablation) | LoRA-MCL · AudioCaps test loss | test loss (NLL): 2.063 -> 1.932 | Improved |
| increase number of hypotheses K from 3 to 7 (ablation) | LoRA-MCL · Clotho test loss | test loss (NLL): 2.663 -> 2.612 | Improved |
| increase LoRA-MLE rank from 8 to 8x7 for parameter parity (ablation) | LoRA-MLE · Clotho test loss | test loss (NLL): 2.812 -> 2.935 | Hurt |
| increase LoRA-MLE rank from 8 to 8x7 for parameter parity (ablation) | LoRA-MLE · AudioCaps test loss | test loss (NLL): 2.203 -> 2.182 | No clear effect |
| replace LoRA-MLE with DBS (tuned lambda) by LoRA-MCL at equal forward passes | LLaVA 1.6 · TextCaps val | SPIDEr: 0.926 -> 0.955 | Improved |
| use LoRA-MCL greedy instead of LoRA-MLE with DBS | LLaVA 1.6 · TextCaps val | mBLEU-4 (lower is more diverse): 0.416 -> 0.520 | Hurt |
| use LoRA-MCL instead of LoRA-MLE with DBS on bilingual synthetic data (French) | LLaVA 1.6 · TextCaps bilingual (French) | SPIDEr: 0.411 -> 0.464 | Improved |
| use LoRA-MCL instead of LoRA-MLE with DBS on bilingual synthetic data (diversity, French) | LLaVA 1.6 · TextCaps bilingual (French) | mBLEU-4 (lower is more diverse): 0.138 -> 0.027 | Improved |
| use LoRA-MCL instead of LoRA-MLE with DBS on bilingual synthetic data (diversity, English) | LLaVA 1.6 · TextCaps bilingual (English) | mBLEU-4 (lower is more diverse): 0.126 -> 0.029 | Improved |
| increase beam size within each group of diverse beam search (ablation) | LoRA-MLE and LoRA-MoE · TextCaps val / Clotho | quality and diversity | No clear effect |
Closest papers by meaning
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.87)
- BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models (NeurIPS 2024 · similarity 0.87)
- Multi-Token Prediction Needs Registers (NeurIPS 2025 · similarity 0.87)
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.86)
- Training-Free Bayesianization for Low-Rank Adapters of Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
