Research map / Multi-agent systems and games / LLM agents

SETA: Scaling Environments for Terminal Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SETA, a pipeline that synthesizes verifiable terminal environments from community Q&A, Kaggle notebooks and NL2Bash (SETA-Synth) and adaptively evolves them by difficulty (SETA-Evol), then trains Qwen3-8B and DeepSeek-V4-Flash with GRPO on the resulting 4,567-environment SETA-Env.

Problem

Scaling RL training environments for terminal agents is hard because tasks lack natural grounding signals, require executable verifiers, and existing synthesis pipelines lack difficulty control.

Approach

Ground terminal task synthesis in human-validated sources via per-source adapter prompts, add a post-rollout trajectory judge that catches unstated-convention test flaws, and adaptively choose difficulty-increase, difficulty-decrease, or context-shift evolution operators per environment based on the training model's pass rate.

Key results

• Best Qwen3-8B SETA (RL) run reaches 12% pass@1 on Terminal-Bench 2.0 vs 3.6% for the best base run (3.3x), per the text; Table reports 10.7±1.3.
• DeepSeek-V4-Flash pass@1 on Terminal-Bench 2.0 improves from 40% to 43.0±2.5% under the CAMEL harness; pass@5 from 54% to 58%.
• CompileBench rises from 6.7% to 40% and CRUST-Bench from 15% to 24% for Qwen3-8B after SETA RL training.

Surprising findings

• Post-rollout trajectory judge flagged about 2% of 100%-failure tasks as design flaws where tests enforced unstated conventions that oracle checks missed.
• Thinking-mode Qwen3-8B produced about 11x more tokens yet matched the non-thinking base score on a case-study task.

Limitations

• Experiments cover only two model families (Qwen3-8B, DeepSeek-V4-Flash) and limited scales and training budgets.
• Environments are limited to terminal-based interaction; no GUI or multimodal settings.
• RL training used a 560-environment subsample filtered for base-model difficulty due to compute constraints.

Experimental setup

Datasets: Stack Overflow Unix & Linux StackExchange Ask Ubuntu Kaggle Notebooks NL2Bash

Benchmarks: Terminal-Bench 1.0 Terminal-Bench 2.0 CRUST-Bench CompileBench QuixBugs

Metrics: pass@1 pass@4 pass@5

Baselines: Qwen3-8B base Qwen3-32B Nemotron-Terminal-8B Endless Terminals-8B OpenThoughts-Agent-v1-RL Qwen3-Coder GPT-5-Nano Claude Haiku 4.5 GPT-OSS-120B Qwen3-30B-A3B

Builds on: GRPO DAPO SWE-bench Self-Instruct Evol-Instruct Harbor

Closest papers by meaning

• On Data Engineering for Scaling LLM Terminal Capabilities (NeurIPS 2026 · similarity 0.89)
• ECHO: Terminal Agents Learn World Models for Free (NeurIPS 2026 · similarity 0.87)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.86)
• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments (ICML 2026 · similarity 0.86)
• TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 · similarity 0.86)
• ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training (ICML 2026 · similarity 0.85)