Research map / Label-efficient and robust learning / Test-time adaptation
The Entropy Enigma: Success and Failure of Entropy Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why entropy minimization (EM) improves accuracy briefly and degrades it over long adaptation, linking this to embedding clustering dynamics, and proposes Weighted Flips, a label-free accuracy estimator built on RDumb adaptation.
Problem
Understanding why entropy minimization helps test-time adaptation for a few steps but fails after many steps, and estimating a classifier's accuracy on an arbitrary unlabeled dataset.
Approach
EM first clusters test embeddings near training class means (success phase) and then drifts them away from training embeddings (failure phase). Accuracy is estimated by counting confidence-weighted prediction flips during EM adaptation (Weighted Flips), mapped to accuracy by a fitted quadratic function.
Key results
- Weighted Flips achieves an average MAE of 5.75% across 23 datasets, versus 8.17% for the prior best COT, a 29.62% relative reduction.
- Removing all initially correctly classified images before Tent adaptation still yields a 10.50% accuracy gain on a holdout set, close to the 12.38% gain without removal.
- During the failure phase, change in Shift distance correlates with accuracy change at rho = -0.79; during the success phase, change in Silhouette score correlates with accuracy at rho = 0.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use weighted flips with quadratic interpolation (ablation) | WF accuracy estimation on ResNet-50 · 23 ImageNet-scale datasets | MAE: 6.14 -> 5.75 | Helped |
| use weighted instead of unweighted flips with quadratic interpolation (ablation) | WF accuracy estimation on ResNet-50 · 23 ImageNet-scale datasets | MAE: 6.27 -> 5.75 | Helped |
| use quadratic instead of linear interpolation for weighted flips (ablation) | WF accuracy estimation on ResNet-50 · 23 ImageNet-scale datasets | MAE: 6.47 -> 5.75 | Helped |
| calculate weighted flips using 100 samples (ablation) | WF accuracy estimation · 23 ImageNet-scale datasets | MAE: 5.75 -> 5.67 | Helped |
| limit adaptation data to 1,000 samples and flips to 100 samples (ablation) | WF accuracy estimation · 23 ImageNet-scale datasets | MAE: 5.75 -> 9.40 | Hurt |
| use Tent instead of RDumb for adaptation (ablation) | WF accuracy estimation on ResNet-50 · ImageNet-scale datasets | MAE: 4.08 -> 5.07 | Hurt |
| use RPL instead of RDumb for adaptation (ablation) | WF accuracy estimation on ResNet-50 · ImageNet-scale datasets | MAE: 4.08 -> 4.97 | Hurt |
| use CPL instead of RDumb for adaptation (ablation) | WF accuracy estimation on ResNet-50 · ImageNet-scale datasets | MAE: 4.08 -> 3.33 | Helped |
| refit the weighted-flips-to-accuracy function on ViT-B/16 data (ablation) | WF accuracy estimation · 10 ImageNet-scale datasets | MAE: 6.61 -> 4.15 | Helped |
| measure weighted flips after 500 rather than 1,000 iterations (ablation) | WF accuracy estimation · ImageNet-scale datasets | MAE: 26.89% | Helped |
| exclude initially correctly classified samples before Tent adaptation (ablation) | Tent · ImageNet-C Gaussian Noise 3 holdout | accuracy gain: 12.38% -> 10.50% | No clear effect |
| exclude samples whose label is outside the top 10 predictions (ablation) | Tent · ImageNet-C Gaussian Noise 3 holdout | accuracy gain: 7.88% | Helped |
Surprising findings
- EM remains effective even when all initially correctly classified images are excluded from adaptation data.
- Using 500 instead of 1,000 iterations for flip counting increases WF performance by a relative 26.89%.
Limitations
- WF is less effective on CIFAR10/100 than on ImageNet-scale datasets, with only 0-5 label flips per dataset on C10-C.
- The weighted-flips-to-accuracy function f must be fit on ImageNet validation and ImageNet-C holdout data, and refitting is needed for some architectures such as ViT-B/16 to get good accuracy.
Experimental setup
Datasets: ImageNet-C ImageNet-Validation ImageNet-C-bar ImageNet-3DCC CCC Stylized ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-D BG Challenge ImageNet-A
Benchmarks: ImageNet-C ImageNet-C-bar ImageNet-3DCC CCC Stylized ImageNet ImageNet-V2 ImageNet-Sketch ImageNet-R ImageNet-D BG Challenge ImageNet-A ImageNet-C Patch
Metrics: mean absolute error silhouette score Spearman correlation
Baselines: AC DoC ATC COT
Builds on: Tent RDumb ETA COT
Closest papers by meaning
- Decoupled Entropy Minimization (NeurIPS 2025 · similarity 0.88)
- Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging (NeurIPS 2026 · similarity 0.87)
- Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification (NeurIPS 2026 · similarity 0.86)
- COME: Test-time Adaption by Conservatively Minimizing Entropy (ICLR 2025 · similarity 0.85)
- Provably Reliable Classifier Guidance via Cross-Entropy Control (NeurIPS 2026 · similarity 0.85)
- Ranked Entropy Minimization for Continual Test-Time Adaptation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.