Research map / Large language models / Agents, code and math
ARBench: Algorithmic Reasoner or API Alchemist? Evaluating LLMs Beyond API Calls
Ren-Biao Liu, Chao-Zeng Ma, Anqi Li, Hui Sun, Xin-Ye Li, Ming Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| go from Easy to Hard tasks on ARB-Model | o4-mini · ARB-Model | Pass@1: 94.29 -> 26.15 | Hurt |
| go from Easy to Hard tasks on ARB-Model | Gemini-2.5-Pro-preview · ARB-Model | Pass@1: 94.29 -> 29.23 | Hurt |
| switch from ARB-Human to ARB-Model split | Deepseek-V3 · ARBench All | Pass@1: 64.84 -> 47.67 | Hurt |
| switch from ARB-Human to ARB-Model split | Qwen2.5-Coder-1.5B-Instruct · ARBench All | Pass@1: 31.25 -> 4.65 | Hurt |
| use reasoning model DeepSeek-R1 instead of DeepSeek-V3 | DeepSeek · ARB-Model | Pass@1: 47.67 -> 59.30 | Improved |
| use reasoning model DeepSeek-R1 instead of DeepSeek-V3 | DeepSeek · ARBench overall | Pass@1: 55.00 -> 60.33 | Improved |
| scale Qwen3 from 14B to 32B | Qwen3 · ARBench overall | Pass@1: 43.67 -> 54.33 | Improved |
| scale Qwen3 from 32B to 235B-A22B | Qwen3 · ARBench overall | Pass@1: 54.33 -> 55.33 | Improved |
| scale Qwen2.5-Coder from 1.5B to 14B | Qwen2.5-Coder · ARBench overall | Pass@1: 16.00 -> 40.67 | Improved |
| use Qwen3-14B instead of Qwen2.5-Coder-14B | 14B model · ARBench overall | Pass@1: 40.67 -> 43.67 | Improved |
| use o4-mini instead of o3-mini | OpenAI reasoning · ARB-Model all | Pass@1: 59.30 -> 65.12 | Improved |
| use o3-mini instead of o4-mini | OpenAI reasoning · ARB-Human all | Pass@1: 67.97 -> 68.75 | Mixed |
Closest papers by meaning
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.88)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.88)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.87)
- Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs? (NeurIPS 2024 · similarity 0.87)
- ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code (NeurIPS 2025 · similarity 0.87)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
