Research map / Label-efficient and robust learning / Self-supervised and contrastive
Cross-Entropy Is All You Need To Invert the Data Generating Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends nonlinear ICA identifiability to DIET (parametric instance discrimination) under a cluster-centric vMF data model and shows cross-entropy supervised classification is a special case that recovers latents up to a linear transformation.
Problem
Why supervised cross-entropy classification learns linearly decodable, interpretable representations; existing identifiability theory covers SSL and auxiliary-variable ICA but not standard supervised classification or the post-training representation.
Approach
Model the data generating process with vMF-distributed latents around unit-norm class cluster vectors; show that minimizing DIET's instance-discrimination cross-entropy identifies latents up to (scaled) orthogonal or affine-linear maps, and that supervised classification is the special case where instance labels are replaced by class labels.
Key results
- In synthetic DIET experiments, R^2 for recovering latents from learned representations reaches about 98-99.8% for most settings (e.g., 99.8% with d=5, 100 classes).
- On DisLib, CNN (ResNet18) representations yield Pearson correlations up to 1.00 for category and 0.99 for posX/posY on dSprites when predicting latents.
- On ImageNet-X, linear decoders from supervised ResNet50 and ViT-B/16 representations decode many proxy factors significantly above chance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train an MLP classifier with cross-entropy instead of decoding from raw inputs | DisLib dSprites · dSprites category | Pearson correlation: 0.26 \scriptscriptstyle\pm 0.00 -> 0.94 \scriptscriptstyle\pm 0.01 | Helped |
| train a CNN classifier with cross-entropy instead of decoding from raw inputs | DisLib Shapes 3D · Shapes 3D objSize | Pearson correlation: 0.21 \scriptscriptstyle\pm 0.00 -> 0.99 \scriptscriptstyle\pm 0.00 | Helped |
| train a CNN classifier with cross-entropy instead of decoding from raw inputs | DisLib MPI 3D · MPI 3D posX | Pearson correlation: 0.28 \scriptscriptstyle\pm 0.00 -> 0.90 \scriptscriptstyle\pm 0.01 | Helped |
| train a CNN classifier with cross-entropy instead of decoding from raw inputs | DisLib MPI 3D real · MPI 3D real posY | Pearson correlation: 0.44 \scriptscriptstyle\pm 0.00 -> 0.71 \scriptscriptstyle\pm 0.02 | Helped |
| use a CNN rather than an MLP encoder | cross-entropy classification on DisLib · DisLib datasets | Pearson correlation | Mixed |
| increase latent dimensionality from 5 to 20 | synthetic supervised classification · Synthetic simulation, |C|=100 | R² (learned latent to ground-truth latent): 99.8\scriptscriptstyle\pm0.00 -> 82.1\scriptscriptstyle\pm0.02 | Hurt |
| use class labels instead of instance indices | cross-entropy classification under the synthetic DGP · Synthetic simulation | R² (learned latent to ground-truth latent) | Helped |
Surprising findings
- Supervised learning with coarse class labels gave higher R^2 than instance-index labels in all but one synthetic setting.
- Identifiability seems to hold even under model misspecification with non-vMF latent distributions.
Limitations
- Experiments mainly on synthetic and controlled datasets; ImageNet-X factors are proxies rather than true latents.
- Assumes vMF latents on hypersphere and conditional independence of augmentations given class, which may not hold in practice.
- Non-Euclidean latents such as orientation are excluded.
Experimental setup
Datasets: DisLib ImageNet ImageNet-X
Benchmarks: DisLib ImageNet-X
Metrics: R^2 MAE of singular values Pearson correlation linear decoding accuracy
Baselines: InfoNCE random initialization raw input decoding
Builds on: DIET generalized contrastive learning time contrastive learning InfoNCE
Closest papers by meaning
- Mechanistic Independence: A Principle for Identifiable Disentangled Representations (ICLR 2026 · similarity 0.88)
- Properties from mechanisms: an equivariance perspective on identifiable representation learning (ICLR 2022 · similarity 0.87)
- Dissecting Supervised Contrastive Learning (ICML 2021 · similarity 0.87)
- Disentangling Factors of Variations Using Few Labels (ICLR 2020 · similarity 0.87)
- Generative Cross-Entropy: A Strictly Proper Loss for Data-Efficient Classification (NeurIPS 2026 · similarity 0.86)
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.