AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

Counterfactual Distillation: Internalizing Reflective Experience into LLM Agent

Rui Ge, Yichao Fu, Yu-Yang Qian, Junda Su, Yiming Zhao, Hao Zhang

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Counterfactual Distillation, which internalizes reflective experience from agent interaction feedback into LLM agents, as stated in a truncated abstract.

Problem

Optimizing LLM agents from long-horizon interaction experience is hard because environmental feedback is heterogeneous and rarely carries explicit reward information, and outcome-driven post-training such as reinforcement learning relies on final outcomes.

Approach

Uses counterfactual reflection on interaction experience and distills the resulting reflective signal into the agent's weights, rather than relying solely on outcome rewards. Details are unavailable because the full text was not provided.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Counterfactual Distillation instead of GRPO-RLVRQwen2.5-7B · WebShopPass@128: 85.40 -> 89.20Improved
use Counterfactual Distillation instead of GRPO-RLVRQwen2.5-7B · SokobanPass@128: 68.00 -> 78.40Improved
use Counterfactual Distillation instead of GRPO-RLVR on Pass@1Qwen2.5-7B · WebShopPass@1: 67.45 -> 66.50Hurt
GRPO->CD instead of GRPO aloneQwen2.5-72B · CodeContestsPass@128: 36.97 -> 47.88Improved
GRPO->CD instead of GRPO alone on Pass@1Qwen2.5-72B · CodeContestsPass@1: 20.45 -> 17.12Hurt
interleave on-policy CD cycles with GRPOQwen2.5-7B · WebShopPass@128: 88.60 -> 91.00Improved
interleave on-policy CD with GRPOQwen2.5-7B · SokobanPass@1: 70.70 -> 78.70Improved
add counterfactual loss L_cf to behavior rehearsal (ablation)Qwen2.5-14B · ScienceWorldPass@128: 67.33 -> 72.00Improved
add counterfactual loss L_cf to behavior rehearsal (ablation)Qwen2.5-7B · ScienceWorldPass@1: 27.67 -> 27.88No clear effect
remove experience from training input (counterfactual context) instead of keeping it (ablation)Qwen2.5-7B · SokobanPass@128: 72.00 -> 78.40Improved
tree-structured rollback branching instead of independent sampling (ablation)Qwen2.5-32B · CodeContestsPass@128: 48.92 -> 55.52Improved
tree branching instead of iterative refinement (ablation)Llama3-70B · CodeContestsPass@128: 38.10 -> 42.50Improved

Limitations

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy