Research map / Vision-language and multimodal / Visual question answering
Breaking Neural Network Scaling Laws with Modularity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a theory showing modular networks on modular tasks have task-dimension-independent sample complexity (unlike exponential for monolithic nets), and proposes a kernel-based rule to align module input projections to the task's modules, validated on sine-wave regression and Compositional CIFAR-10.
Problem
Monolithic networks need sample complexity that grows exponentially with input intrinsic dimensionality, and architectural modularity alone does not reliably exploit task modularity during gradient-based training.
Approach
A linear-regression-on-features theory of NN generalization is extended to modular networks with low-dimensional module projections, showing dimension-independent sample complexity; a kernel-based rule initializes module projections by minimizing the norm of the interpolating module weights, aligning modules to the task's underlying modules before gradient training.
Key results
- Theory predicts monolithic sample complexity grows exponentially with task input dimensionality m, while modular networks on modular tasks have dimension-independent sample complexity.
- On the nonlinear sine-wave variant (k=m=5), the proposed method reaches test loss 0.393 ± 0.099 versus 0.555 ± 0.136 for the baseline modular network and 1.302 ± 0.223 for the baseline monolithic network.
- The learned module initializations align with target module directions (high cosine similarity) and cluster near target modules without any task training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use kernel-based initialization of module projections instead of random initialization | Modular neural network · Nonlinear sine-wave regression (k=m=5) | test loss: 0.555 ± 0.136 -> 0.393 ± 0.099 | Helped |
| Use kernel-based module initialization instead of a randomly initialized monolithic network | Neural network · Nonlinear sine-wave regression (k=m=5) | test loss: 1.302 ± 0.223 -> 0.393 ± 0.099 | Helped |
| Use the proposed modular learning rule rather than random initialization | Modular neural network · Sine-wave regression | required training sample size for desired test error | Helped |
| Use the proposed modular learning rule rather than random initialization | Modular neural network · Compositional CIFAR-10 | accuracy | Helped |
| Initialize module directions with learned directions rather than ground-truth module directions (ablation) | Proposed modular method · Sine-wave regression | generalization performance | No clear effect |
| Allow learned module directions to adapt directly to the task loss (ablation) | Proposed modular method · Modular tasks | performance | Helped |
Surprising findings
- On the sine-wave task, the monolithic baseline outperforms the modular baseline, highlighting optimization difficulty of standard modular architectures.
- Empirically the dependence on task dimension is not eliminated for modular networks, due to optimization difficulty in high dimensions.
Limitations
- Theory assumes linear module projections, linear-in-parameters model, and sum-of-module outputs, which only partially match practical architectures.
- Empirically the task-dimension independence is not achieved because optimization of modular networks is difficult in high dimensions.
- Theory for the monolithic-vs-modular comparison depends on the assumed eigenvalue decay parameterization.
Experimental setup
Datasets: CIFAR-10 Compositional CIFAR-10 (built from CIFAR-10) synthetic sine wave regression
Benchmarks: sine wave regression (synthetic) Compositional CIFAR-10 nonlinear sine wave regression variant
Metrics: test loss sample complexity to reach target test error accuracy cosine similarity between learned and target module directio
Baselines: baseline monolithic network baseline modular network (random initialization)
Builds on: neural tangent kernel Mixture of Experts Compositional MNIST double descent
Closest papers by meaning
- Neural networks are more modular than single neurons suggest (NeurIPS 2026 · similarity 0.90)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.88)
- Discovering modular solutions that generalize compositionally (ICLR 2024 · similarity 0.86)
- Scalable Optimization in the Modular Norm (NeurIPS 2024 · similarity 0.85)
- Range, not Independence, Drives Modularity in Biologically Inspired Representations (ICLR 2025 · similarity 0.85)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.