Research map / Large language models / Agents, code and math
Latent Principle Discovery for Language Model Self-Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STaPLe, a posterior-regularized Monte Carlo EM method where LMs mine latent principles from their own self-correction trajectories, cluster them into a compact constitution, and learn to invoke them for iterative self-improvement.
Problem
Improving open-ended LM generations without human-curated principles or verifiable gold answers; existing constitution-based methods require manual principle curation and static rules.
Approach
Treat self-generated principles as latent variables in a self-correction setting, sample them via rejection sampling using a gold-response proximity score (Rouge-L), train the model on principle-guided refinements in an EM loop, and compress principles via agglomerative clustering with medoid representatives (posterior regularization).
Key results
- STaPLe Iter 4 improves Llama-3.1-8B AlpacaEval from 26.9 to 33.4 (constrained 34.9) and MT-Bench avg from 7.46 to 7.71.
- Across three 7-8B models, STaPLe Iter 4 beats STaR Iter 4 by +1.6-2.8 AlpacaEval points on average and raises MT-Bench turn-2 scores by about 0.22.
- Constrained (clustered) STaPLe largely matches unconstrained performance while shrinking the constitution to roughly 50% of its iteration-1 size by iteration 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Llama-3.1-8B-Instruct · MT-Bench | average score: 7.56 -> 7.71 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Llama-3.1-8B-Instruct · AlpacaEval | win rate: 31.8 -> 33.4 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Llama-3.1-8B-Instruct · IFEval | principle-following win rate: 62.3% -> 68.9% | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Granite-3.1-8B-Instruct · MT-Bench | average score: 7.96 -> 8.04 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Granite-3.1-8B-Instruct · AlpacaEval | win rate: 35.6 -> 38.4 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Granite-3.1-8B-Instruct · IFEval | principle-following win rate: 62.1% -> 67.6% | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Qwen2.5-7B-Instruct · MT-Bench | average score: 7.14 -> 7.24 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Qwen2.5-7B-Instruct · AlpacaEval | win rate: 37.8 -> 40.2 | Helped |
| train with STaPLe principle-guided refinement rather than STaR refined-response SFT | Qwen2.5-7B-Instruct · IFEval | principle-following win rate: 68.4% -> 73.4% | Helped |
| apply agglomerative clustering and use medoid principles (ablation) | STaPLe on Llama-3.1-8B-Instruct · MT-Bench | average score: 7.71 -> 7.70 | No clear effect |
| apply agglomerative clustering and use medoid principles (ablation) | STaPLe on Llama-3.1-8B-Instruct · AlpacaEval | win rate: 33.4 -> 34.9 | Helped |
| apply agglomerative clustering and use medoid principles (ablation) | STaPLe on Granite-3.1-8B-Instruct · IFEval | principle-following win rate: 67.6% -> 68.4% | Helped |
Surprising findings
- Self-Refine without pre-identified principles hurts Llama-3.1-8B on AlpacaEval (26.1 vs 26.9 initial), showing zero-shot self-refinement is not reliably helpful.
- Clustering-based constrained STaPLe sometimes outperforms the unconstrained version on AlpacaEval and IFEval for Llama and Granite.
Limitations
- Focuses on smaller non-reasoning models that struggle with self-refinement; only two-turn self-correction studied.
- Relies on Rouge-L proximity to a gold reference as the correctness signal, which may not capture quality for open-ended tasks.
- Clustering distance thresholds were set manually by the authors after inspecting clusters.
Experimental setup
Datasets: Anthropic HH-RLHF UltraFeedback TL;DR HotpotQA MT-Bench AlpacaEval 2.0 IFEval
Benchmarks: MT-Bench AlpacaEval-2.0-LC IFEval
Metrics: MT-Bench score AlpacaEval win-rate principle-following win-rate Rouge-L F1
Baselines: Self-Refine Gold-only SFT STaR Constrained STaPLe
Builds on: STaR Self-Refine SPIN Constitutional AI SCoRe
Closest papers by meaning
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.91)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.91)
- Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.90)
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.89)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.89)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.