Research map / Large language models / Reasoning and chain of thought

Online Cascade Learning for Efficient Inference over Streams

ICML 2024 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes online cascade learning, which trains a cascade of small models (logistic regression, BERT) online by imitating an LLM expert and learns deferral policies jointly, cutting LLM inference cost by up to 90% while matching LLM accuracy on four stream classification benchmarks.

Problem

LLM inference over data streams is too expensive, and existing cascade and distillation methods require a pre-labeled training set, making them unsuitable for streaming settings.

Approach

Formulates learning a model cascade with deferral policies as an episodic MDP that trades prediction loss against deferral cost, and learns it fully online by imitation of the LLM expert (DAgger-style with decaying jump probability), with calibrated MLP deferral functions and a no-regret OGD guarantee.

Key results

• On IMDB with GPT-3.5 Turbo as LLM, online cascade learning reaches 93.01% accuracy at N=5200 LLM calls vs. 94.15% for GPT-3.5 Turbo, saving about 60% of LLM calls.
• On HateSpeech (N=507), the cascade achieves 82.66% accuracy and 82.36% recall vs. GPT-3.5 Turbo's 83.34% and 83.28%, cutting LLM inference costs by over 90%.
• With Llama 2 70B Chat on FEVER at N=2800, the cascade slightly outperforms the LLM (77.73% vs. 77.15% accuracy).

Measured effects reported in the paper

Surprising findings

• On ISEAR, the cascade with Llama 2 70B Chat surpasses the standalone LLM as cost budget grows, since small models learn from LLM annotations.
• Confidence-based deferral rules are reported to be inadequate for online cascades with noisy LLM annotations, motivating learned deferral functions.

Limitations

• Cascade size must be matched to task complexity; a larger cascade with BERT-large degraded HateSpeech performance.
• Cost model assumes uniform inference cost per model and uses FLOPs estimates rather than measured wall-clock costs for GPT-3.5 Turbo.
• Evaluated only on four binary or small-class text classification benchmarks.

Experimental setup

Datasets: IMDB HateSpeech ISEAR FEVER

Benchmarks: IMDB HateSpeech ISEAR FEVER

Metrics: accuracy recall F1 inference cost (LLM calls)

Baselines: GPT-3.5 Turbo Llama 2 70B Chat Distilled LR Distilled BERT-base Online Ensemble Learning

Builds on: DAgger knowledge distillation online gradient descent

Closest papers by meaning

• C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning (NeurIPS 2025 · similarity 0.90)
• Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.89)
• Cascaded Language Models for Cost-Effective Human–AI Decision-Making (NeurIPS 2025 · similarity 0.88)
• Cascadia: An Efficient Cascade Serving System for Large Language Models (ICLR 2026 · similarity 0.87)
• Online Adaptation of Language Models with a Memory of Amortized Contexts (NeurIPS 2024 · similarity 0.87)
• Language Model Cascades: Token-Level Uncertainty And Beyond (ICLR 2024 · similarity 0.86)