Research map / Label-efficient and robust learning / Recognition and long-tail

On the creation of narrow AI: hierarchy and nonlocality of neural network skills

NeurIPS 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies when narrow neural networks can be trained from scratch (curriculum effects in a compositional sparse parity task) and whether pruning, group-lasso regularization, or distillation better produce small specialized models from large general ones (MNIST and Llama-3.2-1B on Python).

Problem

Creating strong but narrow AI systems: whether narrow skills can be learned from scratch on narrow data, and how to transfer particular skills from large general models into small specialized ones given that skills are distributed across network components.

Approach

Introduces compositional multitask sparse parity (CMSP), where hierarchical composite subtasks are learnable only when atomic subtasks are also in training, and shows that greedy neuron/residual pruning with recovery training, optionally with group-lasso regularization for unlearning, can outperform distillation and training from scratch in data-parameter efficiency.

Key results

• On CMSP, 27/40 networks learned the composite subtask when trained with atomic subtasks, while 0/40 learned it when trained on the composite subtask alone (within 2e9 samples).
• On MNIST even digits, pruning-based methods Pareto-dominate distillation and training from scratch, especially at high neuron counts.
• On Llama-3.2-1B Python pruning with recovery training was more data-efficient than distillation from Llama-3.1-8B or training from scratch, though attribution and random pruning recovered similarly after enough steps.

Measured effects reported in the paper

Surprising findings

• Attribution-based pruning performed about as well as random pruning after moderate recovery training on Llama-3.2-1B.
• Single-hidden-layer networks could still benefit from the curriculum on CMSP, despite the expected need for depth to compute hierarchical features.

Limitations

• CMSP is a contrived toy task, so the curriculum effects may not transfer to natural data.
• Greedy attribution pruning is simple and hyperparameters were not optimally scaled in the LLM experiments, so the comparison may be imperfect.
• Group lasso pruning on MNIST is highly sensitive to hyperparameters.

Experimental setup

Datasets: CMSP synthetic data MNIST GitHub Code Dataset (Python) counterfact AI2-ARC WMDP

Benchmarks: MNIST even digits counterfact AI2-ARC WMDP-Bio WMDP-Cyber

Metrics: test accuracy cross-entropy loss neuron count vs datapoints frontier unlearning success (FT score at least 10% below base)

Baselines: knowledge distillation training from scratch random pruning group lasso pruning

Builds on: multitask sparse parity sparse parity knowledge distillation group lasso

Closest papers by meaning

• The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.87)
• Preparing Lessons for Progressive Training on Language Models (AAAI 2024 · similarity 0.85)
• Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
• Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.85)
• Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.85)
• A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.85)