Research map / Label-efficient and robust learning / Recognition and long-tail

Decoupled Entropy Minimization

NeurIPS 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Learning from unlabeled and positive-unlabeled data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decouples conditional entropy minimization into a reward-driving term (CADF) and a calibrating log-sum-exp term (GMC), then proposes AdaDEM, a hyperparameter-free variant that normalizes CADF rewards and uses a marginal entropy calibrator to beat the tuned upper-bound DEM* across TTA, SSL, UDA and RL.

Problem

Classical entropy minimization (EM) has limited potential: high-confidence samples lose learning signal (reward collapse) and the output distribution becomes biased toward dominant/easy classes (easy-class bias), especially under noisy or dynamic distribution shifts.

Approach

Rewrite the conditional entropy as CADF (-sum p_i z_i, rewards dominant classes) plus GMC (log-sum-exp, penalizes high-confidence logits); add a temperature tau and weight alpha (DEM, with TPE-tuned DEM*), then make it hyperparameter-free in AdaDEM by dividing the CADF gradient by its L1 norm and subtracting an online-estimated class marginal (MEC) in place of GMC.

Key results

• AdaDEM improves Tent single-domain TTA mean accuracy from 52.7% to 66.2% (+13.5), exceeding DEM* (55.1%) without hyperparameter search.
• On CLIP-ViT-B/16 test-time prompt tuning over ImageNet variants, TPT (CoOp) + AdaDEM reaches 65.6% average accuracy vs 64.9% for TPT (CoOp), with +6.5 gain over zero-shot.
• On semi-supervised learning, AdaDEM improves FixMatch on EuroSat by +5.0 and Ent. Min. on CIFAR-100 by +3.2 average accuracy.

Measured effects reported in the paper

Surprising findings

• AdaDEM yields no gain over DeYO on single-domain TTA (+0.0) and slightly lower results than baselines in some SSL cases (e.g., MixMatch on CIFAR-10 -0.2, FixMatch on STL-10 -0.6).
• Reward collapse: classical EM gradient magnitude approaches 0 as max predicted probability approaches 1.0.

Limitations

• DEM* and AdaDEM only modify the objective function and cannot fully solve self-supervised learning problems.
• DEM* requires about 20% labeled test data for hyperparameter search, violating the unsupervised setting.
• Some results are mixed, with AdaDEM matching or underperforming baselines in certain settings (e.g., DeYO, MixMatch, FixMatch on STL-10).

Experimental setup

Datasets: ImageNet-C ImageNet-1K ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch CIFAR-10 CIFAR-100 STL-10 EuroSat TissueMNIST Semi-Aves

Benchmarks: ImageNet-C ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch CIFAR-10 CIFAR-100 STL-10 EuroSat TissueMNIST Semi-Aves GTA5-to-Cityscapes

Metrics: top-1 accuracy mIoU average return F1

Baselines: Tent ETA EATA DeYO SAR TPT Ent. Min. VAT MixMatch FixMatch FreeMatch MinEnt

Builds on: Tent entropy minimization TPE hyperparameter search PPO FixMatch

Closest papers by meaning

• The Entropy Enigma: Success and Failure of Entropy Minimization (ICML 2024 · similarity 0.88)
• Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging (NeurIPS 2026 · similarity 0.85)
• Provably Reliable Classifier Guidance via Cross-Entropy Control (NeurIPS 2026 · similarity 0.85)
• Dynamic Pseudo Labeling via Gradient Cutting for High-Low Entropy Exploration (CVPR 2025 · similarity 0.84)
• A Unified Framework for Entropy Search and Expected Improvement in Bayesian Optimization (ICML 2025 · similarity 0.84)
• COME: Test-time Adaption by Conservatively Minimizing Entropy (ICLR 2025 · similarity 0.84)