Research map / Large language models / Reasoning and chain of thought
Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TREACLE, a DQN-based budget-constrained RL policy that jointly chooses LLM and prompt per question with re-querying, using query embeddings and response-consistency state features, cutting cost up to 85% on GSM8K, CSQA and LLC.
Problem
Answering a stream of reasoning questions with multiple LLMs and prompt styles that differ in accuracy, monetary cost and latency, while respecting a long-term monetary and latency budget and not knowing which option is correct for each question.
Approach
Formulates LLM and prompt selection as a budget-constrained MDP (CMDP) solved by a DQN policy whose state includes question text embedding, response consistency across re-queries, token lengths, re-query counts, and normalized remaining budget; actions are return, re-query, or move to the next LLM-prompt pair in a list sorted by accuracy-to-cost ratio.
Key results
- TREACLE achieves up to 85% cost savings relative to baselines while maintaining high accuracy on GSM8K, CSQA and LLC.
- With a $0.05 budget and beta=50k, TREACLE answers questions with 52.7% correctness versus 23.65% for the cheapest model Llama-2-7b.
- After fine-tuning on 200 GSM8K samples under a CSQA-trained policy, accuracy reaches 0.78 versus 0.848 for a policy trained fully on GSM8K at budget 0.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt per-query latency estimate hourly (historical average) in budget denominator instead of fixed latency (ablation) | TREACLE · GSM8K, time-varying API latency, $0.6 budget | accuracy (%): 86.4 vs 76.1 | Helped |
| Re-querying disabled during training and testing (ablation) | TREACLE and Calibrated Cascade · GSM8K, budget sweep | accuracy | Hurt |
| Ordering LLM-prompt options by accuracy-to-cost ratio in cascade | TREACLE action space · GSM8K | cost minimization (theoretical) | No clear effect |
| Llama-2-7b/13b replaced by fine-tuned MetaMath models | TREACLE · GSM8K, new LLMs | accuracy | Helped |
| Fine-tune DistilBERT accuracy estimator of FrugalGPT on new LLM trajectories | FrugalGPT · GSM8K, new LLMs and prices | accuracy | No clear effect |
| Fine-tune TREACLE on few samples of new LLM options | TREACLE · GSM8K, new cheaper LLM options | adaptation sample count | Helped |
| Re-querying with consistency-aware policy (2-consistency) vs. response consistency feature | TREACLE learned policy · GSM8K, $0.30 budget, alpha=1/20 | fraction of responses 2-consistent | No clear effect |
Surprising findings
- Methods without re-querying eventually match accuracy of re-querying methods but only at significantly larger budgets.
- Learned policy's responses are 93.02% 2-consistent, close to a simple 2-consistent policy.
Limitations
- Focuses on reasoning problems with consistency-based verification; extension to generative tasks requires new consistency measures.
- RL policy budget does not account for the cost of collecting the training query-response data.
- Llama prices are set artificially as fractions (alpha) of GPT prices rather than measured costs.
Experimental setup
Datasets: GSM8K CSQA LLC
Benchmarks: GSM8K CSQA LLC
Metrics: accuracy monetary cost latency
Baselines: FrugalGPT Calibrated cascade Majority Voting Offline knapsack Online knapsack Single model
Builds on: FrugalGPT Deep Q-Network Chain-of-thought prompting AutoMix Mixture of Thought
Closest papers by meaning
- Conformal Constrained Policy Optimization for Cost-Effective LLM Agents (AAAI 2026 · similarity 0.88)
- Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.88)
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.88)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.88)
- SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.88)
- HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.