Research map / Large language models / Retrieval and knowledge

Cascaded Language Models for Cost-Effective Human–AI Decision-Making

NeurIPS 2025 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a cascaded LLM decision framework (small base model, large model, human expert) with calibrated confidence-based deferral and uncertainty-based abstention, with online threshold learning from human feedback.

Problem

Balancing answer correctness, computational cost, and abstention to human experts in LLM question answering, with policies that adapt over time rather than staying static.

Approach

A two-stage cascade where a base LLM's answer is verified with calibrated confidence (Bayesian logistic regression over surrogate token probabilities) and deferred to a larger LLM or to a human based on posterior uncertainty; deferral and abstention thresholds are optimised by SGD on a differentiable system-risk loss using feedback from abstentions.

Key results

• On ARC2-Easy with Qwen-2.5 1.5B→7B, calibrated STP (n=1) verified by the large model reaches ΔIBC of 242.7 (Easy) and 89.3 (Challenge), indicating cost-benefit gains over the large model alone.
• Online cumulative regret of the cascade is lower than single-model baselines on four of five datasets (ARC2-Easy, ARC2-Challenge, MMLU, MedMCQA); MedQA shows no clear gain.
• Verifying with the larger model generally gives better cost-benefit than self-verification, while on medical datasets no method consistently helps (ΔIBC near 0 or negative).

Measured effects reported in the paper

Surprising findings

• Uncalibrated confidence scores sometimes yield higher ΔIBC than calibrated ones, but with much larger standard errors, suggesting instability.
• On the medical datasets (MedQA, MedMCQA), the cascade shows no consistent cost-benefit benefit over the large model alone.

Limitations

• Cascade can collapse to trivial solutions (always using the cheapest or the most accurate model) depending on cost and abstention weights.
• Relies on human feedback that may be sparse or noisy; imperfect expert labels degrade regret.
• Poor confidence estimation on MedQA limits gains; no theoretical guarantees are provided.

Experimental setup

Datasets: ARC-Easy ARC-Challenge MMLU MedQA MedMCQA

Benchmarks: ARC-Easy ARC-Challenge MMLU MedQA MedMCQA

Metrics: Delta IBC (incremental benefit per cost) cumulative regret (system risk) accuracy vs cost per sample

Baselines: Qwen-2.5-1.5B alone Qwen-2.5-7B alone grid search over thresholds

Builds on: Surrogate token probability Platt scaling Monte Carlo dropout Chain-of-thought prompting

Closest papers by meaning

• Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.88)
• Online Cascade Learning for Efficient Inference over Streams (ICML 2024 · similarity 0.88)
• C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning (NeurIPS 2025 · similarity 0.86)
• Cost-efficient Knowledge-based Question Answering with Large Language Models (NeurIPS 2024 · similarity 0.86)
• IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering (NeurIPS 2024 · similarity 0.86)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.85)