Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RL agents trained with annealed access to an oracle or previous generation's behaviour (in-context and in-weights variants) exhibit cultural accumulation, outperforming single-lifetime RL² with equal experience on Memory Sequence, Goal Sequence and TSP.
Problem
Cultural accumulation, the open-ended build-up of knowledge and skills across generations, is under-explored for artificial RL agents, which are typically trained over a single lifetime; existing generational methods rely on hand-crafted imitation learning.
Approach
Train agents with an annealed probability of observing an oracle or the previous generation, with no extra losses, so social learning and independent discovery balance; define in-context accumulation (frozen weights, generations via internal state across trials) and in-weights accumulation (generations as successive training runs).
Key results
- In-context accumulating agents on Memory Sequence outperform single-lifetime RL2 baselines with the same cumulative experience and even exceed the noisy oracles they were trained with.
- In-weights accumulation on Memory Sequence outperforms single-lifetime training for a duration equivalent to 5 complete generations, and resetting combined with accumulation yields higher returns than either alone.
- Performance is non-monotonic in oracle accuracy: oracles that are too accurate or too random degrade in-context accumulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Spread learning across generations rather than one continuous lifetime | In-context accumulation · Memory Sequence (held-out sequence) | return | Helped |
| Spread learning across generations rather than one continuous lifetime | In-context accumulation · Goal Sequence (new goal sequence) | return | Helped |
| Spread learning across generations rather than one continuous lifetime | In-context accumulation · TSP | return | Helped |
| Train with oracle accuracy that is neither too high nor too low (ablation) | In-context accumulation · Memory Sequence | return | Mixed |
| Train with higher, but still imperfect, oracle accuracy (ablation) | In-context accumulation · Goal Sequence | return | Helped |
| Spread training across generations rather than a single continuous run | In-weights accumulation · Memory Sequence | return | Helped |
| Reset the final two feed-forward layers (ablation) | In-weights accumulation · Memory Sequence | return | Helped |
| Combine generational accumulation with resets of the final two feed-forward layers (ablation) | In-weights accumulation · Memory Sequence | return | Helped |
| Spread training across accumulation steps rather than one continuous run | In-weights accumulation · Goal Sequence | return | Helped |
| Spread training across generations rather than one continuous lifetime | In-weights accumulation · TSP | return | Helped |
| Continue in-weights accumulation beyond the initial generations | In-weights accumulation · TSP | return | No clear effect |
Surprising findings
- Oracle accuracy during training has an inverse effect to imitation learning: overly accurate oracles impede accumulation by encouraging over-reliance on social learning.
- Single-lifetime RL2 on TSP with longer episodes failed to make meaningful learning progress.
Limitations
- Environments are small, simple RL tasks (digit sequences, 7x7 gridworld, TSP with 6-24 cities) rather than large-scale domains.
- Selection of the best previous-generation agent is external (oracle selection) rather than learned, with selective social learning only investigated in an appendix.
- Oracle noise and linear annealing of observation probability are hand-set rather than learned.
Experimental setup
Datasets: Memory Sequence (procedurally generated) Goal Sequence (gridworld, procedurally generated) Travelling Salesperson (procedurally generated city positions)
Benchmarks: Memory Sequence Goal Sequence Travelling Salesperson Problem
Metrics: episode return cumulative return across generations compared to single-life
Baselines: RL2 single lifetime (equal episode length to one generation) RL2 single lifetime (equal to all cumulative generations) Kickstarting Iterated policy distillation resetting
Builds on: RL2 PPO S5 PureJaxRL Iterated policy distillation
Closest papers by meaning
- Socialized Learning: Making Each Other Better Through Multi-Agent Collaboration (ICML 2024 · similarity 0.86)
- SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents (NeurIPS 2025 · similarity 0.86)
- MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learning (NeurIPS 2025 · similarity 0.86)
- Bias Amplification in Language Model Evolution: An Iterated Learning Perspective (NeurIPS 2024 · similarity 0.85)
- MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory (NeurIPS 2026 · similarity 0.85)
- CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.