Research map / Multi-agent systems and games / LLM agents
SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SyncMind, a framework for LLM agents' out-of-sync recovery in collaborative software engineering, and SyncBench, a 300-instance evaluation subset (from 24,332 generated instances) built from 21 GitHub Python repositories, measuring success, localization, collaboration willingness, and resource awareness.
Problem
LLM coding agents work in shared, evolving codebases, but their beliefs about the workspace can diverge from the current state ("out-of-sync"), causing failures. Existing benchmarks assume static environments and do not measure recovery, collaboration willingness, or resource awareness.
Approach
Formalizes agent out-of-sync (knowledge gap, state mismatch, task failure) and builds benchmark instances by rolling back repository functions along git history to a failing commit, then evaluates independent vs. collaborator-assisted recovery with LLM-simulated know-everything collaborators and budget/time limits.
Key results
- Claude-3.5-Sonnet reaches 28.18% independent SR, versus 3.33% or lower for Llama-3.1 agents.
- Agents' assistance seeking rates are low, at most 4.86% (Claude-3.5-Sonnet).
- Collaborator assistance improves SR by 0.33% to 5.52%; with full oracle instructions, GPT-4o mini reaches 86.33% SR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| collaborator assistance added to agent recovery | Llama-3.1-8B and other agents · SyncBench | SR: 0.33% to 5.52% gain | Helped |
| collaborator assistance added to agent recovery | DeepSeek · SyncBench Caller | Caller SR: +0.00 | No clear effect |
| collaborator assistance added to agent recovery | Claude-3.5-Sonnet · SyncBench | SR: +5.52% | Helped |
| collaborator assistance added to agent recovery | Claude-3.5-Sonnet · SyncBench Caller | Caller SR: +3.32 | Helped |
| collaborator assistance added to agent recovery | Llama-3.3-70B · SyncBench Callee | Callee SR: +2.00 | Helped |
| collaborator assistance added to agent recovery | GPT-4o · SyncBench Caller | Caller SR: +3.33 | Helped |
| increased task complexity (Callee dependency tracing) vs Caller | most agents · SyncBench | SR: negative Δcomplexity for most agents | Hurt |
| extending turn budget from 30 to 50 turns (ablation) | Llama-3.1-8B · SyncBench time awareness | SR: Independent -0.33%, Collaborative -1.00% | Hurt |
| extending turn budget from 30 to 50 turns (ablation) | Llama-3.1-70B · SyncBench time awareness | SR: Independent +3.67%, Collaborative +4.67% | Helped |
| tripling initial budget ($1000 to $3000) (ablation) | LLM agents · SyncBench budget awareness | SR: [-0.66,+1.67]% | No clear effect |
| halving or doubling cost of assistance seeking (ablation) | LLM agents · SyncBench cost awareness | SR: halved [-2.00,-1.00]%; doubled [-1.00,+1.00]% | No clear effect |
| assistance seeking early in recovery (front-loaded queries) | collaborative agents · SyncBench | SR: first-half assistance seeking 85.71%-100.00% in successes vs 55.76%-97.93% in failures | Helped |
Surprising findings
- Changing financial budgets or assistance costs produces negligible changes in agents' collaboration willingness and success, indicating low resource awareness.
- Tripling the initial budget yields trivial changes in recovery performance.
Limitations
- Benchmark covers only Python GitHub repositories with unit tests, limiting generalizability.
- Out-of-sync simulation from historical commits may not capture live multi-collaborator dynamics.
- Simplified hypothetical resource and cost modeling.
Experimental setup
Datasets: SyncBench (derived from 21 GitHub Python repositories)
Benchmarks: SyncBench
Metrics: success rate localization accuracy conditional success rate assistance seeking rate recovery efficiency
Baselines: Llama-3.1-8B Llama-3.1-70B Llama-3.3-70B DeepSeek-V2.5 GPT-4o mini GPT-4o Claude-3.5-Sonnet
Builds on: OpenHands
Closest papers by meaning
- TOM-SWE: User Mental Modeling For Software Engineering Agents (ICML 2026 · similarity 0.88)
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.88)
- AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.87)
- ProSoftArena: Benchmarking Hierarchical Capabilities of Multi-modal Agents in Professional Software Environments (CVPR 2026 · similarity 0.87)
- Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.87)
- Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.