Research map / Label-efficient and robust learning / Recognition and long-tail

BCE vs. CE in Deep Feature Learning

ICML 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Learning from unlabeled and positive-unlabeled data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically that binary cross-entropy (BCE) also yields neural collapse at its minimum, and empirically that BCE with explicit classifier-bias constraints improves feature compactness and distinctiveness over CE.

Problem

Whether BCE, the multi-class variant decomposing classification into K binary tasks, can induce neural collapse (maximal intra-class compactness and inter-class distinctiveness) like CE, and whether it yields better feature properties in practical training.

Approach

BCE does not satisfy the contrastive property used for prior NC proofs, yet its global minimizers still exhibit NC; BCE measures absolute decision-score values (via sigmoid) and its classifier biases explicitly constrain positive/negative scores across samples, whereas CE only enforces relative ordering and its biases barely matter.

Key results

• On CIFAR100 with AdamW and DA1+DA2, ResNet18 BCE reached 73.2% accuracy vs 71.7% for CE in training setup, and uniform accuracy gains exceeded 20% on CIFAR100 for several models.
• On ImageNet-1k fine-tuning, BCE improved uniform accuracy from 34.48% to 66.92% for ResNet50 (CE 76.74% vs BCE 77.12% top-1).
• On MNIST and CIFAR10, NC metrics of both CE and BCE approach zero, with BCE decreasing faster in the first 20 epochs.

Measured effects reported in the paper

Surprising findings

• Final BCE positive/negative decision scores are separated by the learned classifier bias regardless of bias weight decay, whereas CE biases hardly affect decision scores.
• Higher BCE performance with only basic augmentation (DA1) suggests the gain is not solely due to alignment with Mixup/CutMix, contrary to prior statements.

Limitations

• Theory uses the unconstrained/layer-peeled model and assumes feature dimension d >= K-1; the K > d case is only speculated.
• Experiments are limited to small image benchmarks and ResNet/DenseNet architectures; no Transformer experiments.

Experimental setup

Datasets: MNIST CIFAR10 CIFAR100 ImageNet-1k CIFAR100-LT

Benchmarks: MNIST CIFAR10 CIFAR100 ImageNet-1k CIFAR100-LT

Metrics: classification accuracy uniform accuracy intra-class compactness inter-class distinctiveness NC1 NC2 NC3

Baselines: cross-entropy loss

Builds on: neural collapse unconstrained features model label smoothing loss focal loss

Closest papers by meaning

• Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.86)
• Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model (ICML 2024 · similarity 0.85)
• Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.85)
• Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.85)
• Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.84)
• Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame (NeurIPS 2024 · similarity 0.84)