Research map / Label-efficient and robust learning / Recognition and long-tail
Variational Information Bottleneck for Effective Low-Resource Fine-Tuning
Rabeeh Karimi mahabadi, Yonatan Belinkov, James Henderson
This paper tried a common change: Add dropout
Use dropout regularization · Low-resource GLUE: MRPC, STS-B, and RTE · Accuracy, F1, Pearson, and Spearman: MRPC 87.80 -> 87.33 accuracy; RTE 67.93 -> 65.80 accuracy Hurt
What 795 accepted papers report for this change
- 549 improved
- 87 no clear effect
- 12 mixed
- 147 hurt
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference ICML 2026 · apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately · 20 TPP val loss, 271M · val loss: 2.377 -> 2.367Improved
- Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles ICML 2026 · use MC Dropout · vision and SST-2 · accuracy: 86.3 -> 76.6 on Flowers102Hurt
- Motion Planning in Compressed Representation Spaces ICML 2026 · add nested dropout during training · WOMD guided generation · goal FDE: 4.145 -> 1.104Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a VIB compression bottleneck during fine-tuning | BERT Base · Low-resource GLUE: MRPC, STS-B, and RTE | Accuracy, F1, Pearson, and Spearman: +1.43 accuracy, +2.03 F1, +2.7 Pearson, +2.97 Spearman, +2.6 accuracy | Improved |
| Add a VIB compression bottleneck | BERT · SNLI, 200 training samples | Accuracy: +2.72 | Improved |
| Add a VIB compression bottleneck | BERT · MNLI, 200 training samples | Accuracy: +3.65 | Improved |
| Add a VIB compression bottleneck | BERT · IMDB, 200 training samples | Accuracy: +4.09 | Improved |
| Add a VIB compression bottleneck | BERT · YELP, 200 training samples | Accuracy: +0.7 | Improved |
| Add a VIB compression bottleneck | BERT · SNLI, 6000 training samples | Accuracy: +0.44 | No clear effect |
| Use dropout regularization | BERT Base · Low-resource GLUE: MRPC, STS-B, and RTE | Accuracy, F1, Pearson, and Spearman: MRPC 87.80 -> 87.33 accuracy; RTE 67.93 -> 65.80 accuracy | Hurt |
| Add a VIB bottleneck to suppress task-irrelevant features | BERT trained on SNLI · 15 NLI transfer datasets | Average accuracy improvement over BERT: +5.51 points; improves transfer on 13 out of 15 datasets | Improved |
| Add a VIB bottleneck to suppress task-irrelevant features | BERT trained on MNLI · 15 NLI transfer datasets | Average accuracy improvement over BERT: +3.83 points; improves transfer on 13 datasets | Improved |
| Add VIB regularization to the sentence representation | BERT trained on SNLI · SNLI hypothesis-only probe, test | Accuracy: 53.17 -> 37.10 | Improved |
| Remove the VIB compression loss by setting β=0 (ablation) | VIBERT · MRPC | Accuracy: 89.23 (0.1) -> 88.57 (0.6) | Hurt |
| Remove the VIB compression loss by setting β=0 (ablation) | VIBERT · MRPC | F1: 85.23 (0.2) -> 84.27 (0.7) | Hurt |
Closest papers by meaning
- Visual Instruction Bottleneck Tuning (NeurIPS 2025 · similarity 0.87)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.86)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.86)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima (NeurIPS 2026 · similarity 0.86)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
