Research map / Learning theory and algorithms / Optimal transport
Neural Networks Learn Statistics of Increasing Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that networks trained on real data first perform well on maximum-entropy synthetic data matching low-order statistics and later lose this ability, supporting the distributional simplicity bias across image classifiers and Pythia language models.
Problem
Testing whether neural networks learn low-order data statistics before higher-order ones (the distributional simplicity bias), and extending this to discrete token domains.
Approach
Invert prior DSB setups: train on real data and probe throughout training with synthetic maximum-entropy data that matches only first/second-order statistics (via optimal transport grafting, Gaussian, hypercube-constrained and independent-coordinate sampling), plus prove that n-gram frequencies equal moments of one-hot/embedded sequences.
Key results
- On CIFAR-10, ConvNeXt peaks at over 40% accuracy (w.r.t. target labels) on 2nd-order OT-edited images after 2^10 training steps, well above the 10% chance baseline.
- First-order (ICS) synthetic samples can give some early-training models higher accuracy than real validation images, and accuracy on synthetic data later dips below random baseline (non-monotonicity).
- Pythia loss on unigram and bigram maximum-entropy sequences shows a double-descent pattern, with in-context learning reducing loss later in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Graft target-class means and covariances onto source-class images using optimal transport | Image classification networks · CIFAR-10 target-class labels | accuracy: 10% -> over 40% | Helped |
| Evaluate on first-order independent-coordinate samples instead of second-order Gaussian samples | Image classification networks · Image classification datasets | accuracy | Hurt |
| Evaluate on second-order Gaussian samples instead of first-order independent-coordinate samples | Image classification networks · Image classification datasets | accuracy | Helped |
| Evaluate on first-order synthetic images during training | Image classification networks · SVHN | accuracy | Hurt |
| Train on CIFAR-10 and evaluate on maximum-entropy data matching low-order statistics | Image classification networks · CIFAR-10 | accuracy | Mixed |
| Increase the training step count | Pythia language models · Unigram and bigram sequences | cross-entropy loss | Mixed |
| Evaluate on unigram sequences rather than bigram sequences | Pythia language models · Pile n-gram sequences | cross-entropy loss | Mixed |
| Lengthen the learning-rate warmup period | Pythia 14M and Pythia 70M · Pile n-gram sequences | cross-entropy loss | No clear effect |
| Change the random seed | Pythia language models · Pile n-gram sequences | cross-entropy loss | No clear effect |
| Increase model scale | ConvNeXt V2 and Swin Transformer V2 · Image classification datasets | accuracy | No clear effect |
| Use early-training checkpoints | Image classification networks · MNIST and Fashion MNIST | accuracy | No clear effect |
Surprising findings
- Some early-training models classify independent-pixel (first-order) synthetic samples more accurately than real validation images.
- On SVHN none of the models exceed random-baseline accuracy on first-order synthetic images.
Limitations
- Gaussian optimal transport and hypercube sampling scale poorly with image resolution, limiting experiments to 32x32 or 64x64 images.
- Skip-gram statistics were excluded from the language model experiments for tractability.
- Third-order hypercube sampling is very expensive for CIFARNet, requiring 36 hours on an A40 GPU.
Experimental setup
Datasets: CIFAR-10 Fashion MNIST MNIST SVHN ImageNet-21K The Pile
Benchmarks: CIFAR-10 Fashion MNIST MNIST SVHN CIFARNet Pythia n-gram sequence loss
Metrics: accuracy on synthetic data cross-entropy loss KL divergence
Baselines: Refinetti et al. DSB setup
Builds on: Distributional simplicity bias (Refinetti et al.) Frequency principle Pythia
Closest papers by meaning
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.87)
- How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.86)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
- Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization (NeurIPS 2024 · similarity 0.86)
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.85)
- Learning from higher-order correlations, efficiently: hypothesis tests, random features, and neural networks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.