Research map / Large language models / Agents, code and math
Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a theory-grounded framework and symbol-remapping benchmark for abstract reasoning in LLMs, with metrics Γ (accuracy) and Δ (memory dependence under symbol remapping), and evaluates many LLMs and agent frameworks.
Problem
Existing reasoning benchmarks (ARC, GSM8K, MATH, BIG-Bench Hard) may reward memorization or pattern matching rather than genuine abstract reasoning, and lack a formal grounding for abstraction and reasoning in LLMs.
Approach
Formalizes abstract reasoning as composition of abstraction mapping and rule application, defines Γ (accuracy) and Δ (accuracy drop under systematic symbol remapping) as complementary metrics, and builds a symbolic benchmark of 82 sub-datasets with remapped variants.
Key results
- Number base reasoning (NBR) average Γ is below 0.1 for all 7B and 70B models, while API models reach Γ above 0.5 on decimal tasks.
- Agent framework AgentChat (AutoGen) achieves the highest average Γ of 0.60, but ReAct shows the highest memory dependence (MemDep_all 0.70).
- Fine-tuning Llama-3.1-8B-Instruct on fully remapped data raises the bit-task accuracy from 13% to 60%, but does not transfer to unseen remapping structures; humans reach 97% on that bit task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Chain-of-Thought prompting (ablation) | Qwen2.5-7B-Instruct · Abstract reasoning benchmark (all tasks) | Γ average score: 0.19 -> 0.34 | Helped |
| Use Chain-of-Thought prompting (ablation) | QWQ-32B-Preview · Abstract reasoning benchmark (all tasks) | Γ average score: 0.22 -> 0.50 | Helped |
| Use Chain-of-Thought prompting (ablation) | deepseek v3 · Abstract reasoning benchmark (all tasks) | Γ average score: 0.48 -> 0.41 | Hurt |
| Use Chain-of-Thought prompting (ablation) | gemma-2-9b-it · Abstract reasoning benchmark (all tasks) | Γ average score: 0.18 -> 0.19 | No clear effect |
| Use Chain-of-Thought prompting (ablation) | glm-4-9b-chat · Abstract reasoning benchmark (all tasks) | MemDep_num: 0.26 -> 0.30 | Hurt |
| Fine-tune on fully remapped symbols (ablation) | Llama-3.1-8B-Instruct Direct · fixed_len_chat_bit_dataset | accuracy: 13% -> 60% | Helped |
| Fine-tune on fully remapped symbols (ablation) | Llama-3.1-8B-Instruct Direct · fixed_len_chat_str_dataset | accuracy: 27% -> 25% | Hurt |
| Fine-tune on fully remapped symbols (ablation) | Llama-3.1-8B-Instruct CoT · fixed_len_chat_str_dataset | accuracy: 3% -> 2% | Hurt |
| Use Chain-of-Thought prompting (ablation) | LLMs overall · Math Application and Extended Calculation | Γ accuracy | Helped |
| Use Chain-of-Thought prompting (ablation) | LLMs overall · Number Base Reasoning, Symbolic Math Abstraction, and Symbolic Reasoning | Γ accuracy | No clear effect |
| Apply operand-symbol remapping rather than operator-symbol remapping | Evaluated LLMs · Symbolic Reasoning benchmark | Memory dependence | Hurt |
| Use agentchat(autogen) rather than direct model prompting | gpt-4o-mini · Abstract reasoning benchmark (all tasks) | Γ average score: 0.40 -> 0.60 | Helped |
Surprising findings
- CoT prompting sometimes increases memory dependence Δ, e.g. glm-4-9b-chat's Δ_num rises from 0.26 to 0.30 with CoT.
- Even 70B-scale models show near-zero performance on number base tasks.
Limitations
- Theorems are loosely argued; proofs rely on asymptotic and informal steps rather than rigorous derivations.
- Benchmark tasks are synthetic and small (96 examples per sub-dataset), and results rest on greedy decoding with a single run.
- Results use gpt-4o-mini as the answer parser, which may introduce evaluation error.
Experimental setup
Datasets: GSM8K Symbolic remapped sub-datasets (82 sub-datasets, about 9095 samples)
Benchmarks: Abstract reasoning symbol-remapping benchmark GSM8K
Metrics: Γ (abstract reasoning score) Δ (memory dependence score) accuracy
Baselines: chain-of-thought prompting ReAct AutoGen LLM Debate gemini-2.0-flash-thinking-exp deepseek v3
Builds on: chain-of-thought prompting GSM8K ARC
Closest papers by meaning
- RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation (ICML 2025 · similarity 0.91)
- Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models (ICML 2025 · similarity 0.90)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.90)
- Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning (NeurIPS 2026 · similarity 0.89)
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? (ICML 2025 · similarity 0.88)
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.