Research map / Multi-agent systems and games / LLM agents

Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TerminalTraj, a pipeline that builds 32K Dockerized repo environments, generates Docker-aligned terminal tasks with pytest validation, and produces 50,733 verified trajectories used to SFT Qwen2.5-Coder models that reach 35.30% on TB 1.0 and 22.00% on TB 2.0 at 32B.

Problem

Terminal agent training needs large-scale terminal trajectories that are both executable (suitable Docker environments per instance) and verifiable (standardized checking across heterogeneous task outputs); existing data either lacks execution grounding or has limited environment diversity.

Approach

Filter GitHub repositories with a distilled code-quality reward model, build Dockerfiles into executable environments, synthesize tasks and instance-specific pytest validation code from repo scripts and docs, and keep only trajectories whose final state passes the executable tests.

Key results

• TerminalTraj-32B reaches 35.30% on TB 1.0 and 22.00% on TB 2.0, the best among models under 100B parameters reported in the paper.
• Fine-tuning Qwen2.5-Coder backbones yields gains above +20 points on TB 1.0 and +10 points on TB 2.0 across sizes.
• TerminalTraj-32B reaches 63% pass@16 on TB 1.0, with pass@k rising 20-30% from k=1 to k=16.

Surprising findings

• Code-based verification of trajectories outperforms GPT-5.1-based LLM verification as a filter at 1K-4K training scales.
• Only about 4.92% of generated instances yield verified trajectories, and Docker build success is around 17%.

Limitations

• Verified trajectory yield is low (~4.9%), and Docker build success is only ~17%.
• Training limited to Qwen2.5-Coder up to 32B; larger backbones and RL not explored.
• Automated privacy and license filtering may not have perfect recall.

Experimental setup

Datasets: GitHub repositories (899,741 collected) TerminalBench 1.0 TerminalBench 2.0

Benchmarks: TerminalBench 1.0 TerminalBench 2.0

Metrics: accuracy pass@k 95% bootstrap confidence interval

Baselines: Qwen2.5-Coder-7B-Instruct Qwen2.5-Coder-14B-Instruct Qwen2.5-Coder-32B-Instruct Qwen3-32B-Nex-N1 DeepSeek-V3.1-Nex-N1 Qwen3-Coder-480B-A35B-Instruct Qwen3-235B-A22B-Instruct GPT-5.2 Claude-Opus-4.5 Gemini-3-Pro-Preview DeepSeek-V3.2 GLM-4.7

Builds on: Terminus-2 TerminalBench Megatron-LM OpenHands SWE-bench

Closest papers by meaning

• DOCKSMITH: Scaling Reliable Coding Environments via an Agentic Docker Builder (ICML 2026 · similarity 0.89)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.88)
• TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 · similarity 0.88)
• SETA: Scaling Environments for Terminal Agents (NeurIPS 2026 · similarity 0.86)
• On Data Engineering for Scaling LLM Terminal Capabilities (NeurIPS 2026 · similarity 0.85)
• ECHO: Terminal Agents Learn World Models for Free (NeurIPS 2026 · similarity 0.85)