Research map / Large language models / Agents, code and math

On Data Engineering for Scaling LLM Terminal Capabilities

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic study of terminal-agent SFT data engineering, introducing Terminal-Task-Gen (dataset adapters plus seed-based and skill-based synthetic tasks) and Nemotron-Terminal models fine-tuned from Qwen3 that reach 27.4 on Terminal-Bench 2.0 at 32B.

Problem

Training data strategies behind state-of-the-art terminal agents are undisclosed, and existing approaches (dataset adapters, multi-agent generation) are either limited in structure or too costly to scale for training terminal LLMs.

Approach

A lightweight two-stream pipeline: converting existing math/code/SWE prompts into Terminus 2 format without an LLM, plus LLM-driven synthetic task generation from seeds and a skill taxonomy using pre-built per-domain Docker images instead of per-task Dockerfiles, followed by SFT of Qwen3 models on teacher (DeepSeek-V3.2) trajectories.

Key results

• Nemotron-Terminal-32B reaches 27.4 ± 2.4 on Terminal-Bench 2.0, above Qwen3-Coder-480B (23.9 ± 2.8).
• Nemotron-Terminal-8B reaches 13.0 ± 2.2 versus 2.47 ± 0.5 for Qwen3-8B; Nemotron-Terminal-14B (20.2) beats GPT-OSS-120B (18.7).
• On the 8B ablation, skill-based synthetic tasks alone give 12.4 vs 6.18 for seed-based, and no trajectory filtering beats complete-only (6.74) and success-only (5.06) filtering.

Surprising findings

• Retaining unsuccessful or incomplete trajectories outperforms strict filtering for synthetic tasks.
• Extending context length to 65K with or without YaRN did not help and slightly hurt performance versus the default 32K setting.

Limitations

• Synthetic tasks have no oracle solutions; correctness relies on generated pytest tests.
• Dataset adapter tasks lack test cases, so only unfiltered or incomplete-only filtering can be compared.
• Results are reported with Terminus 2 on Terminal-Bench 2.0 with overlapping standard errors across several comparisons.

Experimental setup

Datasets: Nemotron-Cascade math SFT prompts OpenMathReasoning Nemotron-Cascade code prompts OpenCodeReasoning Nemotron-Cascade SWE prompts SWE-Bench-Train SWE-reBench SWE-Smith SWE-Fixer-Train Terminal-Bench 2.0 (for decontamination)

Benchmarks: Terminal-Bench 2.0 AIME 2024 AIME 2025 LiveCodeBench v6 SWE-bench Verified

Metrics: accuracy (pass@1) mean over runs with std

Baselines: Qwen3-8B Qwen3-14B Qwen3-32B Qwen3-Coder GPT-OSS Gemini 2.5 Flash GPT-5-Mini DeepSeek-V3.2

Builds on: Terminal-Bench Terminus 2 Nemotron-Cascade Evol-Instruct AgentInstruct Harbor

Closest papers by meaning

• SETA: Scaling Environments for Terminal Agents (NeurIPS 2026 · similarity 0.89)
• Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.88)
• TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (NeurIPS 2026 · similarity 0.87)
• It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.87)
• Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.87)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.87)