Research map / Vision-language and multimodal / Multimodal LLMs
TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, Kaipeng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable code interpreter (tool use) (ablation) | o3 · TIR-Bench all tasks | accuracy: 26.9 -> 46.0 | Improved |
| add PyVision thinking-with-images tool framework | GPT-4.1 · TIR-Bench all tasks | accuracy: 18.8 -> 31.8 | Improved |
| enable code interpreter (ablation) | o3 · TIR-Bench Word Search | accuracy: 4.0 -> 64.0 | Improved |
| enable code interpreter (ablation) | o3 · TIR-Bench Rotation Game | accuracy: 33.3 -> 77.3 | Improved |
| enable code interpreter (ablation) | o3 · TIR-Bench Rotated OCR | accuracy: 8.3 -> 53.3 | Improved |
| enable code interpreter (ablation) | o3 · TIR-Bench Proportion VQA | accuracy: 34.2 -> 31.7 | Hurt |
| enable code interpreter (ablation) | o3 · TIR-Bench Jigsaw Game | accuracy: 10.8 -> 16.4 | No clear effect |
| scale open-source model size 3B to 72B | Qwen2.5-VL · TIR-Bench all tasks | accuracy: 17.7 -> 19.7 | No clear effect |
| scale InternVL3 from 8B to 78B | InternVL3 · TIR-Bench all tasks | accuracy: 16.9 -> 21.4 | Improved |
| use explicit prompt to test each degree choice (prompt 3) (ablation) | most models · Rotation Game function calling | accuracy | Improved |
| tool-use SFT instead of direct SFT (ablation) | Qwen2.5-VL-7B · Rotated OCR | accuracy | Improved |
| scale direct SFT data from 1k to 15k (ablation) | Qwen2.5-VL-7B direct SFT · Rotated OCR | accuracy | No clear effect |
Closest papers by meaning
- ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both (NeurIPS 2026 · similarity 0.90)
- When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought (CVPR 2026 · similarity 0.90)
- Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.90)
- CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization (CVPR 2026 · similarity 0.90)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.90)
- Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
