Research map / Large language models / Language, safety and interpretability
Post-Hoc Reversal: Are We Selecting Models Prematurely?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that post-hoc transforms (TS, SWA, ensembling) can reverse performance trends across epochs, model sizes and hyperparameters on noisy data, and proposes post-hoc selection that picks models using post-transform validation metrics.
Problem
Standard practice selects base models by base-model performance before applying post-hoc transforms such as temperature scaling, SWA and ensembling, which can be suboptimal, especially on noisy data where base models overfit.
Approach
Identifies "post-hoc reversal", where post-hoc transforms reverse base performance trends, and proposes post-hoc selection: choose epochs, model sizes and hyperparameters using validation metrics computed after SWA/ensembling/TS rather than base metrics.
Key results
- On FMoW, post-hoc selection gives more than 2x improvement over naive selection in test loss and error, and over 3 points lower test error.
- On an LLM instruction tuning setup (LLaMA-2-7B on Guanaco), post-hoc selection yields more than 1.5x MMLU improvement over naive selection for the SWA+Ens+TS transform.
- Post-hoc selection is better than naive selection in most CIFAR-N and FMoW settings, often with more than 2x improvement; for some clean label sets it is not better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select checkpoints using post-hoc metrics rather than base-model metrics | SWA+TS and SWA+Ens+TS · CIFAR-N and FMoW | test loss and test error: > 2× improvement from post-hoc selection as compared to naive selection | Helped |
| Select checkpoints using post-hoc metrics rather than base-model metrics | SWA+Ens+TS on LLaMA-2-7B · Guanaco test set | MMLU accuracy: > 1.5 × | Helped |
| Use a 0.7-epoch checkpointing interval instead of higher or lower intervals (ablation) | SWA on LLaMA-2-7B · Guanaco instruction tuning | MMLU accuracy | Helped |
| Use post-hoc selection with CE, SOP, or ELR training objectives (ablation) | CIFAR-N models · CIFAR-N | test loss and test error | Helped |
| Replace the CE training objective with SOP or ELR (ablation) | CIFAR-N models · CIFAR-N | test loss and test error | No clear effect |
| Use constant rather than exponentially decaying or cosine learning rates (ablation) | DenseNet121 with post-hoc transforms · FMoW | post-hoc performance | Helped |
| Use SWA and ensembling instead of the base model alone | ResNet18 · CIFAR-10-N noisy label sets | test loss and test error | Helped |
| Use SWA, ensembling, or their compositions | DenseNet121 · FMoW | test error: approx. 10-point improvement | Helped |
| Use SWA instead of ensembling | DenseNet121 · FMoW | test error | Hurt |
| Ensemble SWA models instead of ensembling models alone | DenseNet121 · FMoW | test error | No clear effect |
| Apply TS to align loss with error | Base models · CIFAR-N and FMoW | loss-error mismatch | Helped |
| Use post-hoc selection on OOD rather than ID validation/test splits | SWA+TS and SWA+Ens+TS · FMoW OOD | test loss and test error: > 5 points | Helped |
Surprising findings
- SWA and ensembling favor base models trained for more epochs, even though base models overfit badly early in training.
- Post-hoc transforms can prevent or flatten double descent and mitigate loss-error mismatch.
Limitations
- Post-hoc reversal is less prominent when base curves improve monotonically (low noise, heavy regularization, abundant data), where outcomes match naive selection.
- Post-hoc selection ensembles only models trained for the same number of epochs, so it does not subsume the naive search space and can be marginally worse.
- Evaluation on clean-label-free noisy validation/test sets; results for CE, SOP and ELR losses differ minimally.
Experimental setup
Datasets: CIFAR-10-N CIFAR-100-N FMoW Guanaco Yelp Folktables Reddit-5k Reddit-12k Collab MMLU
Benchmarks: CIFAR-10-N CIFAR-100-N FMoW Guanaco MMLU Yelp Folktables Reddit-5k Reddit-12k Collab
Metrics: test cross-entropy loss test classification error MMLU accuracy perplexity causal language modeling error
Baselines: naive selection
Builds on: stochastic weight averaging temperature scaling double descent effective model complexity
Closest papers by meaning
- A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective (NeurIPS 2025 · similarity 0.85)
- Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy (NeurIPS 2026 · similarity 0.84)
- Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? (ICML 2025 · similarity 0.84)
- Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors (ICLR 2024 · similarity 0.84)
- Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning (NeurIPS 2025 · similarity 0.84)
- A Layer Selection Approach to Test Time Adaptation (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.