Research map / Reinforcement learning / Imitation learning

ECHO: Terminal Agents Learn World Models for Free

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Imitation combined with RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adds a cross-entropy loss on terminal-observation tokens to GRPO's action-token loss (ECHO), reusing the same forward pass, to train CLI agents to predict environment responses and improve pass rates.

Problem

Standard agent RL (GRPO) discards environment observation tokens in multi-turn terminal rollouts, and sparse binary outcome rewards give little signal on failed rollouts.

Approach

Add an auxiliary, length-normalized cross-entropy on non-warning terminal-output tokens to the GRPO loss, using the same logits and rollouts, so the policy learns to predict the environment's responses to its own actions as dense on-policy supervision.

Key results

• On TerminalBench-2.0, ECHO raises Qwen3-8B pass@1 from 2.70% (GRPO) to 5.17%, and Qwen3-14B from 5.17% to 10.79%.
• ECHO sharply lowers env-token cross-entropy on off-policy Qwen3-32B trajectories (e.g., Qwen3-8B val100 0.29 to 0.07) while GRPO barely changes it.
• From base Qwen3-8B, ECHO closes 101.6% of the SFT gap on val100 and about 50% on TB2 pass@1 without expert demonstrations.

Surprising findings

• Env-only training (no reward) improved PyTerm by +10.0pp and ITD by +5.2pp after filtering to clean tool-call rollouts.
• Warning tokens are memorized within ~60 steps, so only terminal-output tokens were used as targets.

Limitations

• Gains on 14B internal evals are smaller, and at 14B ECHO only raises the plateau rather than speeding convergence.
• Env-only verifier-free adaptation degraded TBLite and needed clean-tool-call filtering to help elsewhere.
• TB2 evaluation uses 89 tasks with pass@k at 5 attempts; standard error about 1.5pp, so gaps are noisy.

Experimental setup

Datasets: Endless Terminals OpenThoughts-Agent-v1-RL OpenThoughts-TBLite OpenThoughts-TB-dev TerminalBench 1.0

Benchmarks: val100 ITD OpenThoughts-TBLite TerminalBench-2.0 PyTerm

Metrics: pass@1 pass@3 pass@5 environment-token cross-entropy timeout rate turns completion tokens

Baselines: GRPO OpenThinker-Agent-v1-SFT + GRPO Qwen3-8B base Qwen3-14B base

Builds on: GRPO DAPO Endless Terminals Harbor

Closest papers by meaning

• Learning CLI Agents with Structured Action Credit under Selective Observation (NeurIPS 2026 · similarity 0.88)
• SETA: Scaling Environments for Terminal Agents (NeurIPS 2026 · similarity 0.87)
• RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.87)
• Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.87)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.87)
• Don't Just Fine-tune the Agent, Tune the Environment (ICLR 2026 · similarity 0.87)