Research map / Large language models / Agents, code and math
Investigating Advanced Reasoning of Large Language Models via Black-Box Environment Interaction
Congchi Yin, Tianyi Wu, Yankai Shu, Alex Gu, Yun Han Wang, Jun Shao, xun jiang, Piji Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| extend exploration turns from 10 to 20 and attempts from 1 to 2 (ablation) | 19 LLMs · ORACLE CII, CRI, IPI | average accuracy: over 10% gain | Improved |
| extend exploration turns from 10 to 20 and attempts from 1 to 2 (ablation) | 19 LLMs · ORACLE PSI, ERI, GSI | average accuracy | No clear effect |
| withhold per-turn feedback until the last exploration turn (ablation) | gemini-2.5-pro, o3-mini, o4-mini · ORACLE CRI and ERI 10@1 | accuracy | No clear effect |
| more exploration turns on hard versus easy environments (ablation) | deepseek-v3 · ORACLE hard environments | accuracy gain: near zero for deepseek-v3 vs great improvement for claude-4-sonnet_thinking | No clear effect |
| enable extended thinking (reasoning models) over chat models | LLMs such as claude-4-sonnet · ORACLE | accuracy | Improved |
| raise sampling temperature from 0 to 1 (0.5 for claude) (ablation) | gemini-2.5-flash, deepseek-v3, claude-4-sonnet · ORACLE 20@2&2@1 | accuracy: CRI easy deepseek-v3 +27.1%; IPI easy claude-4-sonnet -10.0% | Mixed |
Closest papers by meaning
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.90)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
