Research map / Multi-agent systems and games / LLM agents

TOM-SWE: User Mental Modeling For Software Engineering Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ToM-SWE, a dual-agent coding framework where a lightweight theory-of-mind agent models user preferences with persistent memory and advises a SWE agent, evaluated on Ambiguous SWE-bench, a new Stateful SWE benchmark, and a three-week developer study.

Problem

Coding agents struggle to infer and track user intent across underspecified instructions and multi-session interactions, and they lack persistent user models.

Approach

A separate ToM partner agent with a three-tier hierarchical memory (raw sessions, session-level user models, overall user model) is consulted in-session for suggestions and updates memory after each session, keeping user modeling out of the SWE agent's context.

Key results

• On Stateful SWE-bench with Claude Sonnet 4, ToMCodeAct resolves 59.7% (text body also reports 57.4%) versus 18.1% for CodeAct.
• On Ambiguous SWE-bench with Claude Sonnet 4, ToMCodeAct achieves 63.4% resolved rate versus CodeAct's 51.9%.
• In a three-week in-the-wild study with 17 developers, 86.2% of 174 ToM suggestions were accepted fully or partially.

Surprising findings

• A single-agent baseline that handles both coding and user modeling reached only 27.17% resolution on Stateful SWE-bench, suggesting context overload.
• RAGCodeAct with Claude 3.7 Sonnet lowered task resolution (18.7% to 14.4%) relative to the plain CodeAct agent, so raw retrieval of past sessions can hurt.

Limitations

• Original 453 developer sessions cannot be released due to privacy, limiting reproducibility.
• Evaluation relies on LLM-based user simulators that may carry systematic biases; the human study is observational without a control group.
• Developer profiles are predominantly English-speaking and may not represent global developer diversity.

Experimental setup

Datasets: SWE-bench Verified Ambiguous SWE-bench 453 consented OpenHands developer sessions

Benchmarks: Ambiguous SWE-bench Stateful SWE-bench

Metrics: task resolved rate user simulator satisfaction score acceptance rate (human study)

Baselines: CodeAct RAGCodeAct

Builds on: CodeAct OpenHands SWE-bench Ambiguous SWE-bench

Closest papers by meaning

• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024 · similarity 0.89)
• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.89)
• SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering (ICML 2025 · similarity 0.88)
• Kimi-Dev: Agentless Training as Skill Prior for SWE-agents (ICLR 2026 · similarity 0.88)
• Interactive Agents to Overcome Underspecificity in Software Engineering (ICLR 2026 · similarity 0.88)
• SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.87)