Research map / Large language models / Agents, code and math
Can LLMs Reason Structurally? Benchmarking via the lens of Data Structures
Yu He, Yingxi Li, Colin White, Ellen Vitercik
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CodeOnly (external interpreter) (ablation) | GPT-4.1 · DSR-Bench code probe | score: 0.40 -> 0.95 | Improved |
| add CodeOnly (external interpreter) (ablation) | Claude-3.5-Sonnet · DSR-Bench code probe | score: 0.44 -> 0.74 | Improved |
| add CodeOnly (external interpreter) (ablation) | Gemini-2.0-Flash · DSR-Bench code probe | score: 0.38 -> 0.44 | No clear effect |
| add CodeEnforce (write code, simulate internally) (ablation) | GPT-4.1 · DSR-Bench code probe | score: 0.40 -> 0.38 | No clear effect |
| add CodeMaybe (optional code) (ablation) | o4-mini · DSR-Bench code probe | score: 0.73 -> 0.76 | No clear effect |
| switch K-D heap tie-breaking from lexicographic order to Euclidean norm (ablation) | o4-mini · DSR-Bench K-D Heap | score: drop of over 0.40 | Hurt |
| use reasoning models instead of instruction-tuned models | DSR-Bench-main · DSR-Bench-main | overall score: 0.79 -> 0.31 | Improved |
| add stepwise or 0-CoT prompts over None prompt (ablation) | instruction-tuned models · DSR-Bench-main | score | Improved |
| add well-designed CoT prompt on suffix tree (ablation) | instruction-tuned models · DSR-Bench Suffix Tree | score | Improved |
| move from formal to realistic narrative prompts | LLMs · DSR-Bench realistic probe | score | Hurt |
| increase dimensionality of spatial data (k = 1, 2, 3, 5) (ablation) | GPT-4.1 / o4-mini · DSR-Bench spatial probe | score | Hurt |
| use non-uniform data distributions (circles, moons, blobs) for K-D tree (ablation) | GPT-4.1 · DSR-Bench spatial probe | score | Hurt |
Closest papers by meaning
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.89)
- Reasoning Structure of Large Language Models (ICML 2026 · similarity 0.89)
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR 2024 · similarity 0.89)
- GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length? (ICML 2025 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
