Research map / Large language models / Agents, code and math
Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies Bayesian iterated learning theory to explain how LLM self-evolution and multi-generation training amplify priors' biases, validated with ICL (ACRE, acronym task) and on-policy DPO experiments.
Problem
Iterative self-data-augmentation and multi-agent LLM interactions may amplify subtle biases over generations, yet there is no systematic behavioral account of why these methods work or how they fail.
Approach
Models LLM agents as Bayesian agents within the iterated learning framework, showing imitation-only evolution converges to the prior's maximum-a-posteriori hypothesis (EM analogy), that interaction phases act as hypothesis filters that steer this convergence, and that in-context sampling approximates Bayesian transmission.
Key results
- Posterior entropy over 243 ACRE rules decreases toward zero across iterated-learning generations, with smaller temperature converging faster.
- With hypothesis-search interaction, GPT-3.5-Turbo (0125) reaches 100% correct d0 predictions and 55% BOTH success versus 0% for imitation-only.
- On-policy DPO on Llama-2-7B amplifies response length well beyond SFT and RLHF baselines; a short-answer constraint in the annotator prompt restrains the length growth.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hypothesis-search interaction instead of imitation-only | LLM agents on ACRE · ACRE | BOTH (%): 15% -> 45% (Old GPT3.5-Turbo 1106); 0% -> 55% (New GPT3.5-Turbo 0125); 30% -> 50% (Claude3-haiku-20240307); 10% -> 30% ( | Helped |
| Add self-refine interaction instead of imitation-only | LLM agents on ACRE · ACRE | BOTH (%): 15% -> 20% (Old GPT3.5-Turbo 1106); 0% -> 35% (New GPT3.5-Turbo 0125); 30% -> 15% (Claude3-haiku-20240307) | Mixed |
| Filter transmitted examples to allow only hard examples | LLM acronym self-data-augmentation · Acronym task, N_e=2 | Ratio-easy: 0.913±0.01 -> 0.219±0.19 (Random -> Hard) | Hurt |
| Filter transmitted examples to allow only hard examples | LLM acronym self-data-augmentation · Acronym task, N_e=2 | Avg-rank: 13519 -> 49869 (Random -> Hard) | Helped |
| Favor easy acronyms with longer lengths | LLM acronym self-data-augmentation · Acronym task, N_e=2 | Avg-length: 5.425±1.04 -> 5.209±0.41 | No clear effect |
| Favor easy acronyms with shorter lengths | LLM acronym self-data-augmentation · Acronym task, N_e=2 | Avg-length: 5.425±1.04 -> 3.975±0.50 | Hurt |
| Increase the initial easy-example count from 2 to 10 | Imitation-only acronym self-data-augmentation · Acronym task | Ratio-easy: 0.438±0.20 -> 0.963±0.00 | Helped |
| Use lower sampling temperature | LLM iterated learning · ACRE | Posterior entropy | Hurt |
Surprising findings
- Strong interaction constraints that make the feasible hypothesis set empty (very-short constraint) significantly harm helpfulness in on-policy DPO.
Limitations
- Theory assumes binary filters, shared priors, and sufficient samples, which real LLM interaction phases violate.
- Experiments are toy-scale artificial tasks (ACRE, acronyms) rather than realistic applications.
Experimental setup
Datasets: Abstract Causal REasoning (ACRE) Anthropic-HH COCA word frequency corpus
Benchmarks: ACRE Anthropic-HH
Metrics: posterior entropy probability of screen:off correct predictions on d0 BOTH ratio ratio of easy acronyms average ranking average response length win rate vs SFT
Baselines: imitation-only iterated learning self-refine hypothesis search random filtering RLHF
Builds on: iterated learning expectation-maximization on-policy DPO self-refine ReST
Closest papers by meaning
- When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings (ICLR 2025 · similarity 0.90)
- Large Language Model Agents Are Not Always Faithful Self-Evolvers (ICML 2026 · similarity 0.88)
- Using Artificial Populations to Study Psychological Phenomena in Neural Models (AAAI 2024 · similarity 0.88)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.87)
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.87)
- Iterated Learning Improves Compositionality in Large Vision-Language Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.