Research map / Large language models / Reasoning and chain of thought
C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C3PO, a label-free LLM cascade that learns per-model confidence thresholds by minimizing regret against the most powerful model, with conformal cost-budget control and PAC-Bayes-style generalization bounds.
Problem
LLM cascades for reasoning usually need labeled data, lack generalization guarantees, and give limited control over test-time inference cost.
Approach
Learn cascade exit thresholds without labels by minimizing 0-1 regret with respect to the most powerful model's answer, using grid search filtered by a conformal quantile check on the cost budget.
Key results
- On MATH-500, C3PO reaches 62.5% accuracy at 0.0019 USD/question versus 57% for self-consistency with the MPM at 0.0053 USD/question.
- Across 300 conformal runs (15 datasets, 2 cascades, 5 budgets, 2 alpha levels), only one cost-budget violation was observed.
- C3PO matches or beats baselines across budgets on the LLaMA cascade, reaching top accuracy at roughly 10x lower cost than other cascade baselines on SVAMP and MATH-500.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use C3PO instead of self-consistency with the MPM | LLAMA cascade · MATH-500 | accuracy: 57% -> 62.5% | Helped |
| Use C3PO instead of self-consistency with the MPM | LLAMA cascade · MATH-500 | cost: 0.0053 USD/question -> 0.0019 USD/question | Helped |
| Use C3PO rather than the compared cascade baselines | LLAMA cascade · MATH-500 difficulty levels | cost | Helped |
| Use C3PO rather than the compared cascade baselines | LLAMA cascade · MATH-500 difficulty levels | accuracy | Helped |
| Train the exit policy on GSM8K and deploy it on MATH-500 | C3PO · MATH-500 | accuracy | Helped |
Surprising findings
- Label-free C3PO outperforms supervised FrugalGPT and TREACLE in the reported comparisons, including under distribution shift from GSM8K/SVAMP to MATH-500.
Limitations
- Conformal cost guarantee can be overly conservative, under-using the budget.
- Grid search scales poorly beyond a handful of models; brute force assumed feasible for fewer than 5 models.
- Generalization bounds assume i.i.d. calibration data and a fixed hypothesis class.
Experimental setup
Datasets: GSM8K SVAMP AQuA CommonSenseQA BIG-Bench-Hard MATH-500 AIME
Benchmarks: GSM8K SVAMP AQuA CommonSenseQA BIG-Bench-Hard MATH-500 AIME
Metrics: accuracy inference cost (USD per question) empirical cost violation rate
Baselines: FrugalGPT Mixture-of-Thoughts (CoT-1D-Vote) TREACLE ModelSwitch self-consistency with MPM
Builds on: FrugalGPT Mixture-of-Thoughts TREACLE ModelSwitch conformal prediction
Closest papers by meaning
- Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.90)
- Online Cascade Learning for Efficient Inference over Streams (ICML 2024 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
- Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (AAAI 2026 · similarity 0.89)
- HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization (AAAI 2026 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.