Research map / Large language models / Agents, code and math

Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents

ICML 2025 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes 1-step lookahead and critic-guided trajectory selection as test-time search methods for non-serializable environments (Docker-based SWE-agent), roughly doubling SWE-bench Verified success for a fine-tuned Qwen2.5-72B policy (40.8%) and improving GPT-4o.

Problem

Test-time search methods such as MCTS cannot be applied to non-serializable RL environments (e.g., Docker containers) where states cannot be saved, restored, or revisited, so LLM agents show high variance across attempts on tasks like SWE-bench.

Approach

Two search operators that need no state rollback: sample-based 1-step lookahead and trajectory selection, both guided by a learned action-value critic (TD(lambda)-trained LLM) that is applied to a policy's sampled actions or complete trajectories.

Key results

• Qwen-based policy with 1-step lookahead and trajectory selection (N=15, K=8) reaches 40.8% success on full SWE-bench Verified, a new open-weights state of the art.
• On Verified-50, Qwen-based policy success rises from 16.2% to 41.7% with lookahead plus trajectory selection (N=10) in the default regime.
• GPT-4o policy success rises from 22.0% to 40.0% with 1-step lookahead plus trajectory selection (N=5) in the default regime.

Measured effects reported in the paper

Surprising findings

• Critic-guided 1-step lookahead causes value hacking, inflating critic values for negative trajectories.
• A GPT-4o critic was not trained on GPT-4o trajectories yet still improved its success rate substantially.

Limitations

• Non-serializable setting precludes MCTS and other rollback-based search, so only forward-only operators are studied.
• Critic quality is bounded: performance saturates at K=8 and the critic is a modest model relative to the GPT-4o policy.

Experimental setup

Datasets: SWE-bench Verified SWE-bench training issues (6,500 issue-PR pairs) bootstrapped agent trajectories (80,000)

Benchmarks: SWE-bench Verified Verified-50

Metrics: success rate standard error of the mean Pass@N

Baselines: Qwen-based policy without search GPT-4o policy SWE-Gym SWE-Fixer Lingma Agent + Lingma-SWE-GPT-72B

Builds on: SWE-agent SWE-bench TD(lambda) Monte-Carlo Tree Search CRIU

Closest papers by meaning

• SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.91)
• Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.88)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.88)
• Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents (ICLR 2026 · similarity 0.87)
• Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.87)
• LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (NeurIPS 2026 · similarity 0.87)