Research map / Multi-agent systems and games / LLM agents
Counterfactual Distillation: Internalizing Reflective Experience into LLM Agent
Rui Ge, Yichao Fu, Yu-Yang Qian, Junda Su, Yiming Zhao, Hao Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Counterfactual Distillation, which internalizes reflective experience from agent interaction feedback into LLM agents, as stated in a truncated abstract.
Problem
Optimizing LLM agents from long-horizon interaction experience is hard because environmental feedback is heterogeneous and rarely carries explicit reward information, and outcome-driven post-training such as reinforcement learning relies on final outcomes.
Approach
Uses counterfactual reflection on interaction experience and distills the resulting reflective signal into the agent's weights, rather than relying solely on outcome rewards. Details are unavailable because the full text was not provided.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Counterfactual Distillation instead of GRPO-RLVR | Qwen2.5-7B · WebShop | Pass@128: 85.40 -> 89.20 | Improved |
| use Counterfactual Distillation instead of GRPO-RLVR | Qwen2.5-7B · Sokoban | Pass@128: 68.00 -> 78.40 | Improved |
| use Counterfactual Distillation instead of GRPO-RLVR on Pass@1 | Qwen2.5-7B · WebShop | Pass@1: 67.45 -> 66.50 | Hurt |
| GRPO->CD instead of GRPO alone | Qwen2.5-72B · CodeContests | Pass@128: 36.97 -> 47.88 | Improved |
| GRPO->CD instead of GRPO alone on Pass@1 | Qwen2.5-72B · CodeContests | Pass@1: 20.45 -> 17.12 | Hurt |
| interleave on-policy CD cycles with GRPO | Qwen2.5-7B · WebShop | Pass@128: 88.60 -> 91.00 | Improved |
| interleave on-policy CD with GRPO | Qwen2.5-7B · Sokoban | Pass@1: 70.70 -> 78.70 | Improved |
| add counterfactual loss L_cf to behavior rehearsal (ablation) | Qwen2.5-14B · ScienceWorld | Pass@128: 67.33 -> 72.00 | Improved |
| add counterfactual loss L_cf to behavior rehearsal (ablation) | Qwen2.5-7B · ScienceWorld | Pass@1: 27.67 -> 27.88 | No clear effect |
| remove experience from training input (counterfactual context) instead of keeping it (ablation) | Qwen2.5-7B · Sokoban | Pass@128: 72.00 -> 78.40 | Improved |
| tree-structured rollback branching instead of independent sampling (ablation) | Qwen2.5-32B · CodeContests | Pass@128: 48.92 -> 55.52 | Improved |
| tree branching instead of iterative refinement (ablation) | Llama3-70B · CodeContests | Pass@128: 38.10 -> 42.50 | Improved |
Limitations
- Full text unavailable; only the truncated abstract was provided.
Closest papers by meaning
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.90)
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.88)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.88)
- Procedural Memory Distillation: Online Reflection for Self-Improving Language Models (NeurIPS 2026 · similarity 0.88)
- Self-evolving LLM agents with in-distribution Optimization (ICML 2026 · similarity 0.88)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
