Research map / Large language models / Agents, code and math
From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics
Bowen Cao, Dongdong Zhang, Yixia Li, Junpeng Liu, Shijue Huang, Chufan Shi, Hongyuan Lu, Yaokang Wu, Guanhua Chen, Wai Lam, Furu Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| embed AIME problems in realistic scenarios (SG) and sub-problem conditions (CS) | Qwen3-32B · AIME 2024 | accuracy: 81.25 -> 67.92 (SG), 57.08 (CS) | Hurt |
| embed AIME 2025 problems in SG and CS scenarios | DeepSeek-R1 · AIME 2025 | accuracy: 86.67 -> 73.33 (SG), 53.33 (CS) | Hurt |
| increase model size within OpenMath-Nemotron series | OpenMath-Nemotron · AIME 2024 CS | relative drop: 77% (1.5B) -> 43% (14B), 51% (32B) | Improved |
| further RL specialization from R1-Distil-Qwen-7B to AceMath-RL-Nemotron-7B | R1-Distil-Qwen-7B · AIME 2024 SG | relative drop: -18% -> -30% | Hurt |
| further training from Qwen3-8B to R1-0528-Qwen3-8B | Qwen3-8B · AIME 2024 CS | accuracy: 42.9 -> 39.6 | Hurt |
| increase problem difficulty from MATH to AIME24 to AIME25 | Qwen3-4B · formulation | formulation accuracy: 84.8 -> 64.6 -> 47.1 | Hurt |
| scale model from Qwen3-0.6B to Qwen3-32B | Qwen3 · formulation | avg formulation accuracy: 42.8 -> 75.0 | Improved |
| add scenario-only SFT instead of original-only SFT (ablation) | Qwen3-8B-Base · ContextMATH AIME 2024 SG | accuracy: 35.4 -> 44.8 | Improved |
| add scenario-only SFT instead of original-only SFT (ablation) | Qwen3-14B-Base · ContextMATH AIME 2024 CS | accuracy: 25.2 -> 38.8 | Improved |
| add scenario SFT to base model (mixture regime) (ablation) | Qwen3-14B-Base · AIME 2024 SG | accuracy: 11.0 -> 52.5 | Improved |
| mix original and scenario data instead of original only (ablation) | Qwen3-8B-Base · average over benchmarks | average accuracy: 52.1 -> 55.6 | Improved |
| scenario SFT on abstract AMC23 (ablation) | Qwen3-8B-Base · AMC23 | accuracy: 83.9 (Ori) -> 83.3 (Syn) | No clear effect |
Closest papers by meaning
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.90)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.89)
- Can LLMs Solve Longer Math Word Problems Better? (ICLR 2025 · similarity 0.88)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.87)
- CORE: Concept-Oriented Reinforcement for Bridging the Definition–Application Gap in Mathematical Reasoning (ICLR 2026 · similarity 0.87)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
