Research map / Label-efficient and robust learning / Recognition and long-tail
Discovering modular solutions that generalize compositionally
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies when modular hypernetworks can discover compositional structure via a teacher-student hyperteacher setup, proves linear identifiability of modules from compositional and connected task support, and shows meta-learning generalizes compositionally on hyperteacher, compositional preference and goal grid worlds.
Problem
When can modular architectures (hypernetworks) discover hidden compositional module structure from task demonstrations and generalize to unseen module combinations, in contrast to monolithic meta-learners?
Approach
Cast compositional generalization as identification of teacher modules in a multi-task teacher-student setting with a linear hypernetwork teacher; prove that compositional and connected task support plus no overparameterization imply the student's modules are a linear transformation of the teacher's, enabling generalization to all module combinations from linearly many sparse combinations.
Key results
- Linear and nonlinear hypernetworks achieve high OOD accuracy on the hyperteacher when task support is compositional and connected, while ANIL and MAML fail to generalize OOD.
- With compositional and connected support, teacher modules are linearly decodable from student task embeddings with R^2 close to one.
- On the compositional preference grid world, hypernetworks achieve lower OOD action-value MSE than ANIL and MAML; in the goal grid world they achieve higher OOD accuracy wrt the optimal policy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use linear or nonlinear hypernetworks instead of ANIL or MAML | Hyperteacher · held-out module combinations with compositional training support | OOD accuracy | Helped |
| use non-compositional training support (ablation) | Hyperteacher hypernetworks · held-out module combinations | OOD accuracy | Hurt |
| train on tasks containing exactly one module per task (K=1) (ablation) | Hyperteacher hypernetworks · held-out module combinations | OOD accuracy | Hurt |
| disconnect the training task support (ablation) | Hyperteacher student · module identification | module alignment | Hurt |
| slightly overparameterize the student (ablation) | Hyperteacher student · module identification | module alignment | Helped |
| increase overparameterization beyond hidden dimension 8h and module dimension 2M (ablation) | Hyperteacher student on discrete training tasks · module identification | module alignment | Hurt |
| use a continuous rather than discrete task distribution while overparameterizing (ablation) | Hyperteacher student · module identification | module alignment | No clear effect |
| use compositional task support rather than holding out a preference vector (ablation) | Compositional preference hypernetworks · grid-world held-out preference combinations | OOD mean-squared error | Helped |
| use linear or nonlinear hypernetworks instead of ANIL or MAML | Compositional preference task · grid-world held-out preference combinations | OOD mean-squared error | Helped |
| cluster training preference vectors into disconnected groups (ablation) | Compositional preference hypernetworks · grid-world held-out preference combinations | OOD loss | Hurt |
| use linear or nonlinear hypernetworks instead of ANIL or MAML | Compositional goal policies · held-out goal compositions | OOD accuracy | Helped |
| hold out one goal quadrant from training (ablation) | Compositional goal hypernetworks · held-out goal compositions | OOD accuracy | Hurt |
Surprising findings
- Hypernetworks with K=1 (one module per task) generalize poorly even with compositional support, because connected support is violated.
- Linear decodability of teacher modules extends to the nonlinear hypernetwork despite architectural mismatch with the teacher.
Limitations
- Theory is in the infinite data limit and assumes no overparameterization, while practice requires slight overparameterization.
- Sample complexity scales unfavorably as the number of teacher modules grows.
- Gradient-based learning dynamics to the modular solution are not characterized.
Experimental setup
Datasets: hyperteacher synthetic tasks compositional preference grid world compositional goal grid world
Benchmarks: hyperteacher OOD tasks compositional preference grid world compositional goal grid world
Metrics: OOD accuracy mean-squared error loss module alignment R^2 of linear decoding
Baselines: ANIL MAML
Builds on: MAML ANIL hypernetworks
Closest papers by meaning
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.88)
- Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks (NeurIPS 2024 · similarity 0.87)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.87)
- Assign and Add: A Mechanistic Study of Compositional Arithmetic (NeurIPS 2026 · similarity 0.87)
- Attention as a Hypernetwork (ICLR 2025 · similarity 0.86)
- CraftFactory: A Conditioned Control Policy Benchmark for Compositional Generalization (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.