Research map / Architectures and efficiency / RNNs and normalization
An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an analytically solvable multilinear model in a skill (orthogonal basis function) representation of multitask sparse parity, deriving scaling laws and calibrating one parameter to predict emergence in 2-layer MLPs and a transformer.
Problem
Understanding emergence of new skills and the origin of neural scaling laws; existing LLM emergence studies are expensive and lack analytic models linking neural dynamics to scaling.
Approach
Represent each skill as an orthogonal basis function on the multitask sparse parity problem with power-law skill frequencies, and model the network as a multilinear expansion a_k b_k g_k; this decouples per-skill dynamics (sigmoidal, Saxe-style) so time, data, and parameter scaling laws and emergence times can be derived analytically, with one calibration constant B.
Key results
- Multilinear model calibrated with a single parameter (B^2=1/22) on one skill predicts the sigmoidal emergence of 5 skills in a 2-layer MLP as training time increases.
- Derived scaling-law exponents: -alpha/(alpha+1) for time and data, -alpha for parameters, -alpha/(alpha+2) for optimal compute.
- Transformer trained on multitask sparse parity (alpha=0.9) shows emergent times following a power law k^(alpha+1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Calibrate the extended multilinear model on the first skill using $\mathcal{B}^2=1/22$ | Multilinear model · Multitask sparse parity, time emergence across 5 skills | skill emergence | Helped |
| Use a $D_c$-shot extension with $D_c=800$ | Extended multilinear model · Multitask sparse parity, data emergence across 5 skills | skill emergence | Helped |
| Use an extended model with $N_c=4$ basis functions per skill | Extended multilinear model · Multitask sparse parity, parameter emergence across 5 skills | skill emergence | Helped |
| Use the multilinear model's single-sigmoid dynamics instead of the neural network's multiple-sigmoid dynamics | Multilinear model compared with 2-layer MLP · Multitask sparse parity, time emergence | time to full saturation | Hurt |
| Use the extended model's decoupled skill dynamics when $d_k \ll D_c$ | Extended multilinear model compared with 2-layer MLP · Multitask sparse parity, data emergence | abruptness of skill emergence | Hurt |
| Use the multilinear model's strict preference for frequent skills | Multilinear model compared with 2-layer MLP · Multitask sparse parity, parameter emergence | consistency of learning less frequent skills | Mixed |
| Use a transformer architecture for time emergence | Transformer · Multitask sparse parity, $\alpha=0.9$ | emergent time scaling with skill index: $\tau_{emerge}(k) \propto k^{\alpha+1}$ | Helped |
Surprising findings
- NN data emergence is more abrupt than the model predicts for skills with few samples, deviating from strict decoupling.
Limitations
- Model cannot capture nonlinear interactions among multiple skills; only linear superpositions.
- NN time emergence has longer saturation tails than the multilinear model because of multiple sigmoidal neuron dynamics.
- Validated only on a synthetic sparse parity setup, not on language tasks.
Experimental setup
Datasets: multitask sparse parity (synthetic)
Benchmarks: multitask sparse parity
Metrics: MSE loss skill strength R_k/S emergence time
Baselines: 2-layer MLP trained on multitask sparse parity linear model with same basis functions
Builds on: Michaud et al. quanta model / multitask sparse parity Saxe et al. two-layer linear network dynamics Hutter feature-based data scaling
Closest papers by meaning
- The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.89)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.88)
- Algorithm Development in Neural Networks: Insights from the Streaming Parity Task (ICML 2025 · similarity 0.87)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.87)
- A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (ICLR 2025 · similarity 0.86)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.