Research map / Architectures and efficiency / RNNs and normalization
Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytically studies how overparameterisation erodes the benefit of curriculum learning in a 2-layer network on an XOR-like Gaussian mixture, with ODE-based online learning and confirmatory experiments on corrupted FashionMNIST and CIFAR10.
Problem
Curriculum learning is theoretically promising but rarely helps deep networks; the paper asks whether high overparameterisation (which also helps via the lottery ticket effect) limits the benefit of curricula.
Approach
Uses the XOR-like Gaussian mixture with a 2-layer ReLU network under online SGD, tracked by ODEs in the high-dimensional limit, and a fading-based curriculum that boosts SNR of a fraction of samples; shows that increasing the number of hidden units shrinks the curriculum gain, providing a theoretical account of its ineffectiveness in deep learning.
Key results
- In the XGM with K=4, curriculum achieves lower noiseless generalisation error than random order and no-fading in a Goldilocks range of noise levels.
- Increasing the number of hidden units K reduces the asymptotic gain of curriculum over no-fading, with the gap vanishing in the highly overparameterised regime.
- On corrupted FashionMNIST with an MLP, the curriculum gain first increases with width up to 100 and then shrinks as overparameterisation grows; on corrupted CIFAR10 the accuracy gap between strategies decreases as filters are increased.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use curriculum instead of random order or no-fading | 2-layer XGM network, K=4 · XOR-like Gaussian Mixture, intermediate SNR | population loss | Helped |
| use curriculum instead of random order or no-fading | 2-layer XGM network, K=4 · XOR-like Gaussian Mixture, high SNR | population loss | Helped |
| use curriculum instead of random order or no-fading | 2-layer XGM network, K=4 · XOR-like Gaussian Mixture, high SNR | cluster coverage | Hurt |
| use curriculum instead of random order or no-fading | 2-layer XGM network, K=4 · XOR-like Gaussian Mixture, low SNR | population loss | No clear effect |
| increase network size | 2-layer XGM network · XOR-like Gaussian Mixture | generalisation error | Helped |
| increase network size in an already overparameterised regime | 2-layer XGM network · XOR-like Gaussian Mixture | curriculum gain over no-fading | Hurt |
| increase MLP width or depth | MLP · corrupted Fashion-MNIST | curriculum gain over random order | Mixed |
| increase the number of convolutional filters | CNN · corrupted CIFAR10 | accuracy gap between curriculum and random order | Hurt |
| apply L2 regularisation to the first layer (ablation) | XGM network · XOR-like Gaussian Mixture, σ=0.4 | performance | Helped |
| apply L2 regularisation to both layers (ablation) | XGM network · XOR-like Gaussian Mixture | performance | Helped |
| vary the curriculum difficulty definition and use continuous or discrete difficulty steps (ablation) | XGM network · XOR-like Gaussian Mixture | curriculum benefit | No clear effect |
| increase network size (ablation) | XGM network trained with a curriculum · XOR-like Gaussian Mixture, very small noise and long training | generalisation performance | Hurt |
Surprising findings
- Curriculum can cause worse performance in the regime of very small noise and large number of epochs, since it reduces fluctuations that help escape sub-optimal basins.
- In the high-SNR regime, curriculum achieves lower loss on average but slightly worse neuron coverage than random order and no-fading.
Limitations
- Theoretical analysis is restricted to the XOR-like Gaussian mixture with a 2-layer network, online learning, and ReLU/logistic loss; generalisation to deep models is conjectured rather than shown.
- Real-data experiments use artificially corrupted FashionMNIST and CIFAR10 with batch learning and tailored curricula, not standard training pipelines.
Experimental setup
Datasets: XOR-like Gaussian mixture (synthetic) FashionMNIST CIFAR10
Benchmarks: XOR-like Gaussian mixture generalisation error corrupted FashionMNIST corrupted CIFAR10
Metrics: generalisation error population loss cluster coverage test accuracy curriculum gap
Baselines: random order no-fading
Builds on: lottery ticket hypothesis XOR-like Gaussian mixture analysis high-dimensional ODE limit of online SGD
Closest papers by meaning
- Position: Lottery Tickets Do Not Explain Overparameterization. How About Escape Dimensions? (NeurIPS 2026 · similarity 0.88)
- Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks (ICLR 2022 · similarity 0.87)
- A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis (ICLR 2023 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.85)
- How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.85)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.