Research map / Architectures and efficiency / RNNs and normalization
Scaling can lead to compositional generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that standard MLPs and transformers compositionally generalize on synthetic compositional task families when data and model size are scaled and training covers the task space, proves an O(1/sqrt(eps)+M) neuron approximation bound, and links linear decodability of task constituents to generalization, including text-to-image composition success.
Problem
Whether standard neural networks (MLPs, transformers) can systematically generalize over tasks sharing compositional structure, and under what data and scale conditions this happens.
Approach
Scaling the number of training tasks and model size makes standard feedforward networks compositionally generalize across task encodings, provided the training support covers modules and is connected; a theorem shows a linear-in-M ReLU MLP can approximate a hyperteacher, and held-out generalization correlates with linear decodability of constituents from hidden activations.
Key results
- Compositional generalization (R²) reaches about 1.00 for identity, orthogonal and language encodings at M=16, K=3 on the hyperteacher, with nonlinear encodings also generalizing given sufficient model size.
- Theorem shows a ReLU MLP approximates a hyperteacher to error eps using O(1/sqrt(eps)+M) neurons, i.e. linear in the number of modules.
- Task-constituent linear decodability from hidden activations correlates with text-to-image generation success across diffusion models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Identity encoding with Orthogonal encoding | MLP · Hyperteacher (M=16, K=3) | compositional generalization R²: 1.00 ± 0.000 -> 1.00 ± 0.000 | No clear effect |
| Use Language encoding | MLP · Hyperteacher (M=16, K=3) | compositional generalization R²: 1.00 ± 0.000 | Helped |
| Use nonlinear Invertible NN encoding | MLP · Hyperteacher (M=16, K=3) | compositional generalization R²: 0.95 ± 0.000 | Helped |
| Use nonlinear Interval shuffle encoding | MLP · Hyperteacher (M=16, K=3) | compositional generalization R²: 0.98 ± 0.010 | Helped |
| Use Few shot task encoding | MLP · Hyperteacher (M=16, K=3) | compositional generalization R²: 0.97 ± 0.001 | Helped |
| Scale the number of training tasks | MLP · Hyperteacher task family | compositional generalization | Helped |
| Increase model size when data is sufficient | MLP · Hyperteacher task family | compositional generalization | Helped |
| Scale training data | Decoder-only transformer · Hyperteacher task family | compositional generalization | Helped |
| Reduce training support coverage by omitting a module | MLP · Hyperteacher task family | compositional generalization | Hurt |
| Make a few modules rare in training | MLP · Hyperteacher task family | compositional generalization | Hurt |
| Increase the frequency of a small set of popular modules | MLP · Hyperteacher task family | compositional generalization | No clear effect |
Surprising findings
- Nonlinear task encodings (invertible NN, interval shuffle, few-shot) still lead to compositional generalization when model size is sufficient.
- Having a small set of popular modules poses no issue, whereas rarely encountered modules cause a drop, suggesting underrepresentation rather than imbalance drives failure.
Limitations
- Theory does not show that SGD will find the linear-in-M generalizing solution.
- Results rely on synthetic data where the generative process and coverage are controlled; real-world training distributions are unknown.
- Focuses on settings where the task is fully specified by the encoding; ambiguous task specification is not studied.
Experimental setup
Datasets: synthetic hyperteacher task family compositional preference grid task family image composition task families (text-to-image prompts)
Benchmarks: hyperteacher held-out compositional tasks compositional preference grid image composition task families
Metrics: R² (compositional generalization) accuracy linear decodability R² of task constituents image generation success rate
Baselines: identity encoding orthogonal encoding language encoding invertible NN encoding interval shuffle encoding few-shot encoding
Builds on: hypernetworks compositional preference task family connected support (hypernetwork teacher-student) invertible neural networks
Closest papers by meaning
- Does Data Scaling Lead to Visual Compositional Generalization? (ICML 2025 · similarity 0.92)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.91)
- Attention as a Hypernetwork (ICLR 2025 · similarity 0.90)
- Scalable Evaluation and Neural Models for Compositional Generalization (NeurIPS 2025 · similarity 0.89)
- Task Generalization with Autoregressive Compositional Structure: Can Learning from $D$ Tasks Generalize to $D^T$ Tasks? (ICML 2025 · similarity 0.89)
- Breaking Neural Network Scaling Laws with Modularity (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.