Research map / Multi-agent systems and games / LLM agents

DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiagEval, a trajectory-conditioned diagnostic protocol that reuses a failed GUI-agent rollout to choose targeted probes and aggregates outcomes into an internal attribution score, reducing false-negative verdicts in LLM-generated web software evaluation.

Problem

GUI-agent evaluators of interactive LLM-generated software observe only one failed trajectory, so failures are ambiguous between evaluator-side execution errors and genuine software defects (single-trajectory identifiability gap).

Approach

Treat post-failure evaluation as sequential active diagnosis: parse the failed trajectory into a Failure Diagnostic Summary, pick a fork node, generate typed diagnostic probes (alternative transitions, observation expansion, reproducibility), rank them by expected information gain, and update a heuristic attribution score with Bayes-style updates, stopping on verified success or an environment-fai

Key results

• On WDJ-U, DiagEval (x2) raises accuracy from 69.9% to 78.3% with Gemini 3 Flash Preview; on RDB from 65.0% to 81.6%.
• On false-negative subsets, DiagEval recovers 45.6-62.1% of failures versus 17.5-46.2% for retry-based baselines.
• DiagEval (x1) with Gemini 3 Flash reaches 76.1% on WDJ-U at $0.122/case, exceeding AppEvalPilot with Claude Opus 4.6 (74.1%) at about 30.5% of its cost.

Surprising findings

• SOU-4 (runtime instability) reproducibility probes produced almost no TN flips on RDB (0.0%), so the gain comes from informative counter-evidence rather than indiscriminate flipping.

Limitations

• Relies on hand-tuned likelihood hyperparameters, so the attribution score is an internal signal rather than a calibrated posterior.
• Depends on an external LLM supervisor (Claude Sonnet 4.6) for FDS construction and branch generation, whose errors propagate.
• Binary attribution space and fail-update assumption P(fail|EnvFail)=1 may be too strong.

Experimental setup

Datasets: WebDevJudge-Unit RealDevBench

Benchmarks: WebDevJudge-Unit RealDevBench

Metrics: accuracy false-negative recovery rate cost per case ROC-AUC entropy reduction

Baselines: AppEvalPilot WebVoyager UI-TARS Agent Model-native CUA Best-of-3 sampling

Builds on: ReAct AppEvalPilot WebVoyager

Closest papers by meaning

• GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions (NeurIPS 2026 · similarity 0.88)
• Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents (ICML 2026 · similarity 0.86)
• Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents (ICLR 2026 · similarity 0.85)
• DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation (NeurIPS 2026 · similarity 0.85)
• OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories (NeurIPS 2026 · similarity 0.84)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents (NeurIPS 2026 · similarity 0.84)