Research map / Large language models / Reasoning and chain of thought
ICPO: Provable and Practical In-Context Policy Optimization for Test-Time Scaling
Tianrun Yu, yuxiao Yang, Zhaoyang Wang, Kaixiang Zhao, Porter Jenkins, Xuchao Zhang, Chetan Bansal, Huaxiu Yao, Weitong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-Context Policy Optimization (ICPO), where an LLM agent improves answers over multi-round self-reflection at inference using self-assessed or external rewards, without parameter updates, with a theoretical account of the mechanism.
Problem
Test-time scaling via multi-round self-reflection: how a model can improve its answers at inference through in-context feedback without modifying its parameters.
Approach
Treats multi-round in-context self-reflection as an in-context policy optimization process driven by self-assessed or externally observed rewards, and explains it theoretically via pretraining under a Fisher-weighted log-likelihood objective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ME-ICPO multi-round in-context self-improvement | Qwen2.5-Math-7B · AIME 2024 | Mean@16 (%): 11.04 -> 30.42 | Improved |
| apply ME-ICPO | Qwen2.5-Math-7B · AMC | Mean@16 (%): 41.42 -> 47.06 | Improved |
| apply ME-ICPO | Qwen2.5-Math-1.5B · AIME 2024 | Mean@16 (%): 6.46 -> 9.79 | Improved |
| apply ME-ICPO | Qwen2.5-Math-1.5B · MATH-L1 | Mean@16 (%): 49.27 -> 61.19 | Improved |
| apply ME-ICPO | Qwen2.5-Math-7B · MATH-L5 | Mean@16 (%): 30.58 -> 38.71 | Improved |
| add minimum-entropy response selection (ablation) | ME-ICPO (Qwen2.5-Math-7B) · AIME 2024 | Accuracy (%): 5.77 -> 30.05 | Improved |
| add explicit reward tags in context (ablation) | ME-ICPO (Qwen2.5-Math-7B) · AIME 2024 | Accuracy (%): 19.30 -> 30.05 | Improved |
| add reward without entropy selection (ablation) | ME-ICPO (Qwen2.5-Math-7B) · AIME 2024 | Accuracy (%): 5.77 -> 6.21 | No clear effect |
| use ground-truth reward instead of self-assessed majority vote (ablation) | ME-ICPO (Qwen2.5-Math-7B) · AIME 2024 | Accuracy (%): 30.05 -> 38.19 | Improved |
| apply majority voting on top of ME-ICPO outputs | ME-ICPO · AIME 2024 | Maj@k | Improved |
Closest papers by meaning
- SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning (ICML 2026 · similarity 0.88)
- Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (ICML 2025 · similarity 0.87)
- Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.87)
- Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.87)
- What If We Allocate Test-Time Compute Adaptively? (ICML 2026 · similarity 0.87)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
