Research map / Reinforcement learning / Visual and representation RL
Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a teacher-student RL perceptron model of compositional tasks and uses statistical-physics ODEs to show that primitives pre-training (shaping) speeds learning and improves noise robustness over vanilla training.
Problem
Shaping procedures in animal training can determine whether a complex task is learned quickly, slowly, or not at all, yet no theory explains their role or guides their design in compositional task learning with policy gradient methods.
Approach
Models compositional RL tasks as linear blends of K primitive RL-perceptron teacher-student pairs gated by a context vector, derives asymptotically exact ODEs for online policy-gradient-like learning, and analyses primitives pre-training versus vanilla training to explain shaping benefits (faster learning, noise robustness).
Key results
- Primitive learning timescale scales as 2^(T-2)/(eta_w(T-2)), exponential in episode length T.
- Primitives pre-training gives substantial speed-ups over vanilla training, with the benefit larger for higher T and lower initial context alignment V0V*.
- Primitives pre-training is more robust to noise in context updates (sigma_v) than vanilla training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use primitives pre-training instead of vanilla training | Compositional RL task · Synthetic compositional task | training time | Helped |
| Use primitives pre-training instead of vanilla training when initial context alignment is poor | Compositional RL task · Synthetic compositional task | training time | Helped |
| Increase episode length T | Vanilla training and primitives pre-training · Synthetic compositional task | training time | Hurt |
| Increase initial context alignment with the target context | Primitives pre-training · Synthetic compositional task | compositional generalization training time | Helped |
| Increase a primitive's initial context alignment with its target context | Vanilla training · Synthetic compositional task | primitive learning time | Helped |
| Increase the initial alignment of one primitive | Vanilla training · Synthetic compositional task | learning time of the other primitive | Helped |
| Use primitives pre-training instead of vanilla training when context-update noise is relatively large and primitive-update noise is small | Compositional RL task · Noisy synthetic compositional task | learning performance: σ_w≤0.05, σ_v≥0.05 | Helped |
| Use primitives pre-training instead of vanilla training when primitive-update noise is large | Compositional RL task · Noisy synthetic compositional task | learning performance: σ_w≥0.05 | No clear effect |
Surprising findings
- Learning one primitive accelerates learning of the other primitives (multiplicative effect), even though primitives are modular and independent.
Limitations
- Limited to linear combinations of sub-tasks with strictly modular, orthogonal primitives.
- Spherical-case analysis and high-dimensional limit (N to infinity) assumptions.
- Only sequential-free compositional structure; other compositional forms (chaining, recursive) not analysed.
Experimental setup
Metrics: generalisation error order parameters R and V training time
Baselines: vanilla training
Builds on: RL perceptron REINFORCE teacher-student framework
Closest papers by meaning
- Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · similarity 0.88)
- Separating the 'what' and 'how' of compositional computation to enable reuse and continual learning (NeurIPS 2025 · similarity 0.86)
- How does RL Post-training Induce Skill Composition? A Case Study on Countdown (NeurIPS 2026 · similarity 0.86)
- Deep Reinforcement Learning with Time-Scale Invariant Memory (AAAI 2025 · similarity 0.86)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.86)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.