Research map / Large language models / Reasoning and chain of thought

Conformal Constrained Policy Optimization for Cost-Effective LLM Agents

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CCPO, which combines constrained policy optimization (CPO with V-trace off-policy correction) and online conformal prediction to learn a cost-aware orchestration policy over a cheap base LLM and a GPT-4o guide that meets a user-specified coverage level at lower API cost.

Problem

Cost-effective LLM agent orchestration that minimizes API cost while guaranteeing, via conformal prediction, that the prediction set covers the correct answer whenever some action sequence can produce it.

Approach

Parameterize a conformal policy as thresholding a stochastic policy's probabilities (set = actions with pi >= kappa), train a smooth stochastic surrogate with V-trace importance-weighted CPO updates, and jointly adapt the threshold kappa with online conformal prediction to guarantee coverage.

Key results

• On HotpotQA with LLaMA-2-7B (alpha=0.1), CCPO (lambda=0) reaches cost 6.552 cents at coverage 0.902 versus 7.835 for CPO batch and 7.484 for CPO online, the 12-27% cost reduction over CPO variants cited in the paper.
• On MMLU with LLaMA-2-7B (alpha=0.1), CCPO (lambda=0) costs 7.949 cents at coverage 0.920 versus 22.25 for CPO batch and 19.07 for CPO online.
• Pointwise LLM-guided and RL baselines such as CPO without conformal calibration often fail to reach the target coverage (e.g., CPO coverage 0.70 on MMLU at alpha=0.1).

Surprising findings

• Pointwise CPO often fails to reach target coverage at higher coverage levels, and LLM-as-policy approaches with small base models perform near random.
• GPT-4o EXIT incurs cost of 827 cents on HotpotQA versus under 10 cents for the learned policies.

Limitations

• Coverage guarantee relies on the policy converging for the asymptotic result; batch calibration is needed for finite-sample guarantees.
• Evaluated only on two QA benchmarks with 200 test examples and a small number of base/guide model pairs.
• Cost reductions are reported relative to certain baselines, and set sizes can be larger than those of pointwise CPO.

Experimental setup

Datasets: HotpotQA MMLU

Benchmarks: HotpotQA MMLU

Metrics: cost (cents) coverage average episode length prediction set size

Baselines: Random policy GPT-4o LLaMA-2-7B LLaMA-3.2-3B LLM-EXIT UALA CPO CPO batch CPO online

Builds on: CPO V-trace (IMPALA) UALA adaptive conformal inference FrugalGPT

Closest papers by meaning

• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.90)
• One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.88)
• VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.88)
• Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning (NeurIPS 2024 · similarity 0.88)
• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.88)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.88)