Research map / Multi-agent systems and games / LLM agents

DOCKSMITH: Scaling Reliable Coding Environments via an Agentic Docker Builder

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains DockSmith, a 30B-A3B Docker-building agent via SFT on execution-grounded SWE-Factory-style trajectories with loop detection and cross-task memory, reaching SOTA on Multi-Docker-Eval and transferring gains to SWE-bench and Terminal-Bench.

Problem

Docker-based environment construction for repositories is brittle and is a bottleneck for scaling execution-grounded training and evaluation of software engineering agents.

Approach

Treat Docker environment construction as a core agentic capability: extend SWE-Factory with loop-detection and cross-task success memory to generate cleaner trajectories, then fine-tune Qwen3-Coder-30B-A3B on curated, complexity-balanced Docker-building trajectories jointly mixed with SWE-solving data.

Key results

• DockSmith (30B-A3B) reaches 39.72% F2P and 58.28% Commit Rate on Multi-Docker-Eval, a +20.3 F2P gain over its base Qwen3-Coder-30B-A3B-Instruct (19.46%).
• Mixing Docker-building trajectories at a 1:0.5 SWE:Docker token ratio yields +3.37 on Terminal-Bench 2.0 over SWE-only training.
• Docker-building errors drop from 3,757 to 2,161 total (-42.5%) relative to the baseline on Multi-Docker-Eval.

Surprising findings

• Docker-only training yields weak SWE benchmark performance (SWE.V 33.50), while adding SWE-solving data improves all benchmarks, suggesting the two data types are complementary.
• Upsampling long-tail languages indiscriminately degraded performance in preliminary trials, so per-language token limits were used instead.

Limitations

• Gains on SWE benchmarks are modest (about +2 points on SWE.V and SWE.M at best) and mixing ratio effects are non-monotonic on Terminal-Bench.
• Error analysis relies on GPT-5.1 automated annotation without reported human validation.
• Context Retrieval errors increase by 10.9% and env-layer resolution rate drops, indicating trade-offs.

Experimental setup

Datasets: Multi-Docker-Eval Nex Agent-SFT GitHub merged pull requests (in-house crawl)

Benchmarks: Multi-Docker-Eval SWE-bench Verified SWE-bench Multilingual Terminal-Bench 2.0

Metrics: Fail-to-Pass (F2P) Commit Rate average input tokens average output tokens average Docker image count resolve rate

Baselines: DeepSeek-v3.1 DeepSeek-R1 GPT-OSS-20B GPT-OSS-120B Kimi-K2-0905 Kimi-K2-thinking Qwen3-235B-A22B Claude-Sonnet-4 GPT-5-Mini Gemini-2.5-Flash Qwen3-Coder-30B-A3B-Instruct

Builds on: SWE-Factory Multi-Docker-Eval Qwen3-Coder

Closest papers by meaning

• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments (ICML 2026 · similarity 0.89)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.87)
• MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering (ICML 2026 · similarity 0.87)
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.87)
• RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.87)
• daVinci-Dev: Agent-native Mid-training for Software Engineering (ICML 2026 · similarity 0.86)