Research map / Large language models / Agents, code and math
PILOT-Bench: Probabilistic Interaction for LLM Operations in Tool-driven Scenarios
Ruicheng Ao, Zeping Min, Tingyu Zhu, Wotao Yin, Xinshang Wang
This paper tried a common change: Chain-of-thought prompting
add Chain-of-Thought instructions to baseline prompt · ResiliBench simulated · full success rate (%): 51.4 -> 50.8 No clear effect
What 1,107 accepted papers report for this change
- 765 improved
- 94 no clear effect
- 52 mixed
- 196 hurt
- AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing ICML 2026 · add structured chain-of-thought reasoning to the diffusion LLM · StoryGen-Eval editing · delta multiFLAM: 27.8 -> 11.7Improved
- Vision Language Models Cannot Reason About Physical Transformation ICML 2026 · add CoT prompt instead of Direct prompt · Number & Length · accuracy: F(3, 333) = 18.28Hurt
- VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation ICML 2026 · Add CoT cold-start stage on top of SFT · Long-RVOS · J&F: 47.12 -> 51.82Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace baseline prompt with MDP-optimal workflow prompt | 7 LLMs average · ResiliBench simulated | full success rate (%): 51.4 -> 62.1 | Improved |
| add Chain-of-Thought instructions to baseline prompt | 7 LLMs average · ResiliBench simulated | full success rate (%): 51.4 -> 50.8 | No clear effect |
| add Chain-of-Thought instructions | GPT-4o-mini · ResiliBench simulated | full success rate (%): 50.5 -> 56.1 | Improved |
| add Chain-of-Thought instructions | Llama-3.3-70B · ResiliBench simulated | full success rate (%): 47.8 -> 43.6 | Hurt |
| inject systematic flaws into the MDP-optimal workflow prompt | 7 LLMs average · ResiliBench simulated | full success rate (%): 62.1 -> 54.3 | Hurt |
| inject flaws into workflow prompt | Gemini-2.5-Flash · ResiliBench simulated | full success rate (%): 60.1 -> 20.0 | Hurt |
| inject flaws into workflow prompt | GPT-4o-mini · ResiliBench simulated | full success rate (%): 67.7 -> 62.2 | Hurt |
| inject flaws into workflow prompt | GPT-5-mini · ResiliBench simulated | full success rate (%): 60.7 -> 63.5 | No clear effect |
| inject flaws into workflow prompt | DeepSeek-V3 · ResiliBench simulated | full success rate (%): 56.8 -> 58.4 | No clear effect |
| scale Qwen2.5 from 3B to 7B | Qwen2.5 with MDP-optimal prompt · ResiliBench simulated | full success rate (%): 0.5% -> 63.5% | Improved |
| scale Qwen2.5 from 32B to 72B | Qwen2.5 with MDP-optimal prompt · ResiliBench simulated | full success rate (%): 65.0% -> 65.0% | No clear effect |
| inject flaws into workflow prompt | 7 LLMs average · ResiliBench real-world task set | full success rate (%): 46.7 -> 34.8 | Hurt |
Closest papers by meaning
- WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models (ICLR 2025 · similarity 0.86)
- Reducing Tool Hallucination via Reliability Alignment (ICML 2025 · similarity 0.86)
- MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.85)
- ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.84)
- Do LLMs estimate uncertainty well in instruction-following? (ICLR 2025 · similarity 0.84)
- MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
