Research map / Large language models / Agents, code and math
From Winning to Understanding: A Diagnostic Long-Horizon RTS Benchmark for LLMs
Jiacheng Li, Jiahui Liu, Yuqing Wang, Gaochen Cui, Xiao Zhang, Qianchuan Zhao, Ziyou Zhang, Chenghao Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add structured strategic guidance and full state description to the prompt (ablation) | GPT-4o · Prompt sensitivity vs rule-based AI | win rate: 0.15 / 0.85 / 0 -> 0 / 0.59 / 0.41 (W/D/L, standard vs simplified) | Improved |
| replace precise coordinates with ambiguous directional cues | DeepSeek · Track C Encirclement | task score: 0.600 -> 0.221 | Hurt |
| replace precise coordinates with ambiguous directional cues | Claude · Track C Encirclement | task score: 0.647 -> 0.746 | No clear effect |
| replace precise coordinates with ambiguous directional cues | Gemini · Track C Encirclement | task score: 0.386 -> 0.357 | Hurt |
| replace precise coordinates with ambiguous directional cues | Kimi · Track C Encirclement | task score: 0.495 -> 0.380 | Hurt |
| evaluate in cross-play against adaptive LLM opponents instead of scripted rule-based AI | GPT-4o · Track A vs Track B ranking | ranking: win rate 0.45 (Track A, top macro throughput) -> Elo 1602.90 (third) | Hurt |
| evaluate in cross-play against adaptive LLM opponents instead of scripted rule-based AI | Claude · Track A vs Track B ranking | ranking: win rate 0.70 -> Elo 2136.02 | Improved |
| add the Scouting metric to a four-metric regression on win rate in Track B | win-rate regression · Track B | standardized beta: Scouting 0.238 vs MU -0.152 | Improved |
| use MU alone as predictor of win rate versus jointly with KLR, MC and Scouting | win-rate regression · Track A | correlation: r = 0.95 -> beta = 0.012 | Mixed |
| increase number of sampled games per pair in subsampling from 2 to 3 | Track B ranking · Track B subsampling | exact ranking recovery: 80.8% -> 93.5% | Improved |
Closest papers by meaning
- TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.88)
- Fast and Accurate Probing of In-Training LLMs' Downstream Performances (NeurIPS 2026 · similarity 0.87)
- Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs (NeurIPS 2025 · similarity 0.86)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.85)
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.85)
- Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
