Research map / Large language models / Language, safety and interpretability
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dropout and weight decay to zero with mixout toward pretrained weights | BERT-LARGE · RTE, MRPC, CoLA, STS-B | mean dev score: 56.5 -> 64.0; 83.4 -> 89.0; 38.8 -> 57.9; 82.4 -> 89.6 | Helped |
| Combine mixout toward pretrained weights with weight decay toward pretrained weights | BERT-LARGE · RTE, MRPC, CoLA, STS-B | mean dev score: 65.3; 87.8; 58.3; 89.7 | Mixed |
| Add mixout toward the initialization of the additional output layer (ablation) | BERT-LARGE with mixout 0.7 on pretrained layers · RTE, CoLA, STS-B | mean dev score: 61.6 -> 66.5; 57.4 -> 58.7; 89.6 -> 89.7 | Helped |
| Add mixout toward the initialization of the additional output layer (ablation) | BERT-LARGE with mixout 0.7 on pretrained layers · MRPC | best dev score: 91.1 -> 92.4 | Hurt |
| Use dropout 0.7 instead of mixout on the additional output layer (ablation) | BERT-LARGE with mixout 0.7 on pretrained layers · MRPC | best dev score: 91.1 -> 92.5 | Helped |
| Use mixout instead of dropout on the additional output layer (ablation) | BERT-LARGE with mixout 0.7 on pretrained layers · RTE, CoLA, STS-B | mean dev score: 57.2 -> 66.5; 48.9 -> 58.7; 89.2 -> 89.7 | Helped |
| Increase mix probability across a broad range (ablation) | BERT-LARGE mixout toward pretrained weights · RTE | mean dev score | Helped |
| Increase dropout probability (ablation) | BERT-LARGE dropout · RTE | mean dev score | Hurt |
| Use mixout plus pretrained-weight decay on a larger training set (ablation) | BERT-LARGE · SST-2 | mean dev score: 93.4 -> 93.5 | No clear effect |
| Apply mixout toward pretrained weights during MNIST finetuning instead of dropout | 784-300-100-10 network pretrained on EMNIST Digits · MNIST and EMNIST Digits | validation accuracy: 0.992 to approximately 0.723 | Mixed |
| Apply mixout toward pretrained weights instead of dropout | BERT-LARGE · RTE | unusable finetuning runs: 15 ... among 20; 4 ... out of 20 | Helped |
| Add mixout toward pretrained weights to stabilize finetuning | BERT-LARGE · RTE, MRPC, CoLA, STS-B | test score: 70.1 -> 70.2; 89.3 -> 89.1; 60.5 -> 62.1; 86.5 -> 87.3 | Mixed |
Closest papers by meaning
- Model Merging in Pre-training of Large Language Models (NeurIPS 2025 · similarity 0.87)
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.87)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.87)
- Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws (NeurIPS 2026 · similarity 0.86)
- SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization (NeurIPS 2026 · similarity 0.86)
- AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.