Research map / Large language models / Reasoning and chain of thought
Adaptive Social Learning via Mode Policy Optimization for Language Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASL, a framework that trains language agents with four cognitively-inspired reasoning modes (behavioral cloning then RL) and AMPO, an advantage that combines mode-level and sample-level terms to switch reasoning depth adaptively in social interactions.
Problem
Language agents in social intelligence simulations either lack explicit reasoning or use uniformly long chain-of-thought, causing excess token use and inflexible social behavior in negotiation and collaboration tasks.
Approach
Four hierarchical reasoning modes (intuitive response to prospective deduction) are injected via behavioral cloning with mode control tokens; AMPO adds a mode-level advantage (mode reward, or length when rewards tie) to a GRPO-style sample-level advantage so the policy learns context-aware mode selection and shorter reasoning.
Key results
- AMPO on Llama3.1-8B reaches GOAL 8.06 on SOTOPIA-Hard vs 6.97 for GPT-4o (self-play setting numbers as reported in the text, described as a 15.6% gain).
- AMPO outperforms GRPO on SOTOPIA-Hard (3.68 vs 3.44 overall with Llama) while using 581 vs 865 tokens per turn (about 32.8% fewer).
- Removing the length reward raises Qwen2.5-7B AMPO average tokens from 647 to 1617 with lower goal score on SOTOPIA-Hard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GRPO advantage with AMPO mode-level + sample-level advantage | Qwen2.5-7B-Instruct (BC-initialized) · SOTOPIA-Hard self-play | GOAL: 7.44 -> 7.85 | Helped |
| Replace GRPO advantage with AMPO mode-level + sample-level advantage | Llama3.1-8B-Instruct (BC-initialized) · SOTOPIA-Hard self-play | OVERALL: 3.44 -> 3.68 | Helped |
| Replace GRPO advantage with AMPO; reduce tokens | Llama3.1-8B-Instruct · Avg tokens per turn | Avg Tokens: 865 -> 581 tokens (lower) | Helped |
| Add behavioral cloning on four reasoning modes before RL | Qwen2.5-7B-Instruct · SOTOPIA-Hard self-play | GOAL: 6.21 -> 7.14 (BC alone) | Helped |
| Add answer length reward r^l (ablation) | AMPO (Qwen2.5-7B-Instruct) · Avg tokens | Avg Tokens: 1617 -> 647 tokens with r^l | Helped |
| Add answer length reward r^l (ablation) | AMPO (Qwen2.5-7B-Instruct) · SOTOPIA-Hard self-play | GOAL: 7.56 -> 7.85 | Helped |
| Add answer length reward r^l (ablation) | GRPO (Qwen2.5-7B-Instruct) · Avg tokens | Avg Tokens: 1705 -> 905 tokens with r^l | Helped |
| Use single mode M1 (intuitive response) instead of hybrid modes (ablation) | Qwen2.5-7B-Instruct · SOTOPIA-Hard self-play | GOAL: 7.85 -> 7.08 | Hurt |
| Add hybrid four-mode reasoning under GRPO (vs mode-free) (ablation) | GRPO (Qwen2.5-7B-Instruct) · SOTOPIA-Hard self-play | OVERALL: 3.16 -> 3.41 | Helped |
| Remove diverse state data (keep only fixed-turn data) (ablation) | AMPO (Qwen2.5-7B-Instruct) · SOTOPIA-Hard self-play | GOAL: 7.85 -> 7.12; tokens 647 -> 139 | Hurt |
| Remove goal-aware marginal-contribution reward (ablation) | AMPO (Qwen2.5-7B-Instruct) · SOTOPIA-Hard self-play | OVERALL: 3.54 -> 2.91; tokens 647 -> 726 | Hurt |
| Replace hybrid modes with TALE-PT / LS-Mix / O1-Pruner efficient reasoning methods | Qwen2.5-7B-Instruct · SOTOPIA-Hard self-play | GOAL: AMPO 7.85 vs TALE-PT 6.21, LS-Mix 6.96, O1-Pruner 6.91 | Hurt |
Surprising findings
- Large reasoning models such as o1, o3-mini, DeepSeek-R1 and QwQ-32B underperform non-reasoning proprietary models on social goal completion.
- GRPO-trained models tend to collapse toward the most complex reasoning mode (M4), while AMPO keeps diverse modes active.
Limitations
- RL training requires substantial compute and hardware resources.
- Inference time increases due to generating reasoning tokens, though fewer than GRPO.
- Policy optimization uses single-turn training on multi-turn social interactions, raising long-horizon consistency questions.
Experimental setup
Datasets: SOTOPIA-pi SOTOPIA SOTOPIA-Hard DEMO NegotiationArena
Benchmarks: SOTOPIA SOTOPIA-Hard DEMO NegotiationArena
Metrics: GOAL OVERALL average tokens self profit total profit winning rate
Baselines: GPT-4o Claude-3.5-Sonnet DeepSeek-V3 OpenAI-o1 DeepSeek-R1 QwQ-32B PPDPP EPO DAT DSI GRPO TALE-PT
Builds on: GRPO PPO SOTOPIA Hierarchical Cognitive Control Theory
Closest papers by meaning
- A$^2$FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning (ICLR 2026 · similarity 0.89)
- ALSO: Adversarial Online Strategy Optimization for Social Agents (ICML 2026 · similarity 0.88)
- R-4B: Incentivizing General-Purpose Auto-Thinking in MLLMs via Bi-Mode Annealing and Reinforce Learning (CVPR 2026 · similarity 0.87)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.87)
- Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective (ICML 2026 · similarity 0.86)
- LEAD: Length-Efficient Adaptive and Dynamic Reasoning for Large Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.