Research map / Label-efficient and robust learning / Recognition and long-tail
Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models
Guande He, Jianfei Chen, Jun Zhu
This paper tried a common change: Add label smoothing
Add label smoothing to JL-P · SWAG · ECE: 1.94 -> 1.20 Improved
What 190 accepted papers report for this change
- 127 improved
- 19 no clear effect
- 7 mixed
- 37 hurt
- Anti-Backdoor Coreset Selection via Cumulative Entropy ICML 2026 · add unlearning step with label smoothing · CIFAR10 WaNet · poisoning rate of coresetImproved
- When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models ICML 2026 · add label smoothing to BT loss · RewardBench Reasoning · accuracy: 75.41 -> 72.28Hurt
- Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision–Language Models ICML 2026 · apply label smoothing instead of plain CE · Average of 9 datasets, 40% sym noise · accuracy (%): 78.09 -> 78.39Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add joint MLM training on downstream text (JL-D) (ablation) | Full fine-tuning with RoBERTa-base · SNLI/MNLI, QQP/TwitterPPDB, SWAG/HellaSWAG | ECE: 2.21_{0.12} -> 0.67_{0.05} (SNLI); 4.06_{0.20} -> 1.26_{0.50} (MNLI); 2.44_{0.10} -> 0.57_{0.12} (QQP); 10.09_{0.49} -> | Improved |
| Use pre-training-phase corpus instead of downstream text for the MLM objective (JL-P) (ablation) | Joint learning · MNLI, TwitterPPDB, HellaSWAG | ECE: 1.26 -> 1.20 (MNLI); 6.49 -> 1.27 (TwitterPPDB); 7.64 -> 5.23 (HellaSWAG) | Improved |
| Use pre-training-phase corpus for MLM instead of downstream text (ablation) | Joint learning · SWAG | ECE: 0.91 -> 1.94 | Hurt |
| Add label smoothing to JL-P (ablation) | JL-P with pre-trained-model distillation · SWAG | ECE: 1.94 -> 1.20 | Improved |
| Add pre-trained-model knowledge distillation to JL-D (ablation) | JL-D · QQP/TwitterPPDB | ECE: 0.57_{0.12} -> 0.50_{0.08} (QQP); 6.49_{0.44} -> 6.06_{0.52} (TwitterPPDB) | Improved |
| Add pre-trained-model knowledge distillation to JL-P (ablation) | JL-P · SWAG/HellaSWAG | ECE: 2.85_{0.11} -> 1.94_{0.12} (SWAG); 7.64_{0.37} -> 5.23_{0.43} (HellaSWAG) | Improved |
| Add pre-trained-model knowledge distillation to JL-P (ablation) | JL-P · SNLI | ECE: 1.12_{0.17} -> 1.09_{0.15} | Improved |
| Add pre-trained-model knowledge distillation to JL-P (ablation) | JL-P · QQP | ECE: 0.78_{0.47} -> 0.91_{0.50} | Hurt |
| Add contextualized-representation L2 regularization (ablation) | JL-D · ID and OD settings | ECE | Improved |
| Increase the MLM loss weight (ablation) | Joint learning · NLU tasks | accuracy | Hurt |
Closest papers by meaning
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.90)
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.89)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.88)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective (NeurIPS 2024 · similarity 0.87)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
