Research map / Large language models / Reasoning and chain of thought
Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM cascades that route questions from GPT-3.5-turbo to GPT-4 based on answer consistency across mixed Chain-of-Thought and Program-of-Thought samples, reaching GPT-4-level accuracy at about 40% of its cost.
Problem
Reducing the API cost of using strong LLMs for reasoning tasks by deciding which questions a weaker, cheaper LLM can answer without invoking the stronger model.
Approach
Uses the weaker LLM's answer consistency as a training-free difficulty signal for routing, with vote-based and verification-based decision rules, and mixes CoT and PoT thought representations so that consistent answers across different reasoning formats indicate high confidence.
Key results
- MoT cascade variants reach about 0.929 average accuracy versus 0.931 for GPT-4-CoT-SC while costing about 40% of its cost.
- On CREPE, MoT variants outperform GPT-4-CoT-SC (0.885 vs. 0.871) at 47% of its cost.
- CoT-2D-Vote achieves 0.924 average accuracy at 57% relative cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add mixture of thought (CoT + PoT samples) to vote-based cascade (ablation) | GPT-3.5 weaker LLM cascade (MoT-1D-Vote vs CoT-1D-Vote) · GSM8k | accuracy at comparable cost: 0.951 vs 0.929 at ~0.4 relative cost | Helped |
| Use two sets of CoT demonstrations instead of one for vote-based cascade (ablation) | CoT-2D-Vote vs CoT-1D-Vote · GSM8k | accuracy: 0.945 vs 0.929 at 0.65/0.70 thresholds (~0.4 avg. cost gain reported) | Helped |
| Verification-based MoT (CoT+PoT) cascade reaches near GPT-4 accuracy at lower cost | MoT-1D-Verify · GSM8k | accuracy vs relative cost: 0.947 accuracy at 0.401 relative cost vs GPT-4 CoT-SC 0.958 at 1.000 | Helped |
| Use stronger LLM verification with weaker-LLM answer hints (ablation) | MoT-1D-Verify with hints to GPT-4 · GSM8k and CREPE | accuracy: large drop reported on GSM8k and CREPE; slight gain on DATE | Hurt |
| Replace GPT-3.5 with LLaMA2-13B as weaker LLM (ablation) | Cascade approaches · GSM8k and CREPE | accuracy: not stated numerically; described as not yielding ideal results | Hurt |
| Use external fine-tuned RoBERTa or GPT-3.5 verifiers instead of answer consistency (ablation) | Cascade decision maker · GSM8k | accuracy: best external verifier 0.892 vs 0.951 for proposed approaches | Hurt |
Surprising findings
- External fine-tuned verifiers and LLM-based verifiers fail to reach GPT-4-CoT-SC accuracy on complex reasoning tasks, e.g., 0.892 vs 0.958 on GSM8K.
- Passing the weaker LLM's inconsistent answers as hints to GPT-4 greatly hurts performance on GSM8K and CREPE.
Limitations
- LLaMA2-13B as weaker LLM performs poorly on GSM8K and CREPE, so the weaker model must be capable enough on the task.
- Cost comparisons use coarse-grained alignment and simplifications rather than exact costs.
- DATE lacks a training set, so the second demonstration set was manually annotated by the authors.
Experimental setup
Datasets: GSM8K ASDIV TabMWP DATE Navigate CREPE
Benchmarks: GSM8K ASDIV TabMWP DATE Navigate CREPE
Metrics: task accuracy relative cost
Baselines: GPT-3.5-CoT-SC GPT-3.5-PoT-SC GPT-4-CoT-Greedy GPT-4-PoT-Greedy GPT-4-CoT-SC GPT-4-PoT-SC Finetuned-Q Finetuned-QA LLM-Q LLM-QA
Builds on: chain-of-thought prompting program-of-thought prompting self-consistency FrugalGPT
Closest papers by meaning
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.91)
- C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning (NeurIPS 2025 · similarity 0.90)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)
- A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.89)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
- Online Cascade Learning for Efficient Inference over Streams (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.