Research map / Architectures and efficiency / Pruning and efficient tuning
Better Fine-Tuning by Reducing Representational Collapse
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, Sonal Gupta
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add R3F regularization during RoBERTa-Large fine-tuning | RoBERTa-Large · GLUE development tasks | task metrics (accuracy or Matthews correlation) | Improved |
| Replace adversarial SMART regularization with parametric-noise R3F regularization | RoBERTa-Large · GLUE development tasks | task metrics (accuracy or Matthews correlation) | Mixed |
| Add R3F regularization to XLM-R Large fine-tuning | XLM-R Large · XNLI test zero-shot | average accuracy across languages: 80.9 -> 81.2 | Improved |
| Add R4F regularization, including spectral normalization of the classification head | XLM-R Large · XNLI test zero-shot | average accuracy across languages: 80.9 -> 81.4 | Improved |
| Add R3F regularization to BART fine-tuning | BART · CNN/DailyMail | ROUGE-1: 44.16 -> 44.38 | Improved |
| Add R3F regularization to BART fine-tuning | BART · Gigaword | ROUGE-1: 39.29 -> 40.45 | Improved |
| Add R3F regularization to BART fine-tuning | BART · Reddit TIFU Long | ROUGE-1: 24.19 -> 30.31 | Improved |
| Use trust-region-style regularization instead of standard fine-tuning | RoBERTa fine-tuned on SST-2, then probed on other GLUE tasks · GLUE probing experiments | probe performance | Improved |
| Use R3F or R4F rather than standard fine-tuning through sequential task updates | RoBERTa · Sequential GLUE probing | source-task probe performance | Improved |
Closest papers by meaning
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.87)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.87)
- LoFiT: Localized Fine-tuning on LLM Representations (NeurIPS 2024 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.87)
- Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
