Research map / Large language models / Agents, code and math
TypedThinker: Diversify Large Language Model Reasoning with Typed Thinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TypedThinker, which uses a fine-tuned meta-thinker to predict effective reasoning types (deductive, inductive, abductive, analogical) per problem plus retrieved type-specific demonstrations to guide an LLM reasoner, improving logical and math reasoning.
Problem
LLM reasoning typically relies on deductive step-by-step reasoning, and diverse reasoning types are hard to exploit because the right type must be identified per problem and the model must actually follow it.
Approach
A meta-thinker is fine-tuned to predict empirical effectiveness scores (success rate over sampled solutions) for each reasoning type; a memory of rejection-sampled, reverse-checked type-specific solutions supplies retrieved demonstrations, and a fine-tuned reasoner executes the chosen type.
Key results
- TypedThinker with self-consistency @5 averages 0.422 on Mistral 7B vs 0.388 for few-shot + SC @5, and 0.562 on LLaMA3 8B vs 0.497 for few-shot + SC @5.
- Single-response TypedThinker improves Qwen2-7B average accuracy from 0.521 (few-shot) to 0.596.
- Using the Mistral 7B meta-thinker with GPT-4o, TypedThinker + SC @5 reaches 0.81 on LogiQA and 0.90 on BBH versus 0.80 and 0.85 for GPT-4o + SC @5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TypedThinker instead of Few-shot | Mistral 7B · LogiQA, BBH, GSM8K, and MATH (average accuracy) | accuracy: 0.318 -> 0.364; +SC @5: 0.388 -> 0.422 | Helped |
| use TypedThinker instead of Few-shot | LLaMA3 8B · LogiQA, BBH, GSM8K, and MATH (average accuracy) | accuracy: 0.428 -> 0.481; +SC @5: 0.497 -> 0.562 | Helped |
| use TypedThinker instead of Few-shot | Qwen 2-7B-Instruct · LogiQA, BBH, GSM8K, and MATH (average accuracy) | accuracy: 0.521 -> 0.596; +SC @ 5: 0.598 -> 0.668 | Helped |
| add self-consistency voting over 5 responses | TypedThinker on Mistral 7B · LogiQA | accuracy: 0.554 -> 0.570 | Helped |
| add self-consistency voting over 5 responses | TypedThinker on LLaMA3 8B · GSM8K | accuracy: 0.585 -> 0.753 | Helped |
| add weighted voting over the effective reasoning types instead of selecting the best type and using SC @5 | TypedThinker, Mistral 7B · LogiQA | accuracy: 0.570 -> 0.581 | Helped |
| add weighted voting over the effective reasoning types instead of selecting the best type and using SC @5 | TypedThinker, Mistral 7B · BBH, MATH | accuracy: 0.469 -> 0.453; 0.149 -> 0.127 | Hurt |
| remove fine-tuning from the reasoner (ablation) | TypedThinker, Mistral 7B · LogiQA, BBH, GSM8K, and MATH | accuracy: -0.076; -0.041; -0.102; -0.018 | Hurt |
| remove the meta-thinker and use CoT Selection (ablation) | TypedThinker, Mistral 7B · LogiQA, BBH, GSM8K, and MATH | accuracy: -0.025; -0.036; -0.152; -0.024 | Hurt |
| replace retrieved demonstration memory with human-written few-shot examples (ablation) | TypedThinker, Mistral 7B · LogiQA and BBH | accuracy: -0.082; -0.051 | Hurt |
| replace retrieved demonstration memory with human-written few-shot examples (ablation) | TypedThinker, Mistral 7B · MATH | accuracy: 0.013 | Helped |
| add reasoning types to few-shot sampling | Mistral 7B · LogiQA, BBH, GSM8K, and MATH | Levenshtein distance: 0.573 -> 0.644; 0.599 -> 0.650; 0.498 -> 0.710; 0.659 -> 0.732 | Helped |
Surprising findings
- CoT Selection chose 'none' over 60% of the time and, when choosing a type, mostly chose deductive, which was effective on only 34% of problems.
- Removing the explicit demonstration memory improved MATH performance for Mistral 7B, as retrieved math solutions with different numbers misled the reasoner.
Limitations
- A single problem may need different reasoning types at different steps, which a single type per problem cannot capture.
- Evaluation is limited to logical and mathematical benchmarks; code and creative tasks are not studied.
- GPT-4o and MetaMath results use random 100-example subsets for GPT-4o, limiting statistical precision.
Experimental setup
Datasets: LogiQA BBH GSM8K MATH Contexthub LiveBench
Benchmarks: LogiQA BBH GSM8K MATH Contexthub LiveBench
Metrics: accuracy Levenshtein distance n-gram overlap Kendall's tau
Baselines: Few-shot CoT Selection Self-Discover Zero-shot MoR Few-shot MoR
Builds on: self-consistency Self-Discover MetaMath
Closest papers by meaning
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.90)
- SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks (AAAI 2025 · similarity 0.89)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
- THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.89)
- RMath: A Logic Reasoning-Focused Datasets Toward Mathematical Multistep Reasoning Tasks (AAAI 2025 · similarity 0.88)
- RATT: A Thought Structure for Coherent and Correct LLM Reasoning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.