Research map / Large language models / Alignment and preferences
Beyond Step Pruning: Information Theory Based Step-level Optimization for Self-Refining Large Language Models
Jinman Zhao, Erxue Min, Hui Wu, Ziheng Li, Zexu Sun, Hengyi Cai, Shuaiqiang Wang, Xu Chen, Gerald Penn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| step-level DPO instead of solution-level DPO | Qwen2.5-7B · Math average | average accuracy: 52.5 -> 53.7 | Improved |
| prune steps with ISLA score (12% data) vs all steps (ablation) | Step-DPO Qwen2.5-7B · Math average | average accuracy: 53.7 -> 55.0 | Improved |
| prune steps with ISLA score (12% data) vs all steps (ablation) | Step-DPO Qwen2.5-3B · Math average | average accuracy: 45.5 -> 46.1 | Improved |
| prune steps with ISLA score (20% data) vs all steps (ablation) | Step-DPO Qwen2.5-1.5B · Math average | average accuracy: 38.4 -> 39.2 | Improved |
| combine relative accuracy and logp instead of logp only (ablation) | ISLA step selection · Math average (Qwen2.5-7B) | average accuracy: 52.9 -> 55.0 | Improved |
| combine relative accuracy and logp instead of relative accuracy only (ablation) | ISLA step selection · Math average (Qwen2.5-7B) | average accuracy: 54.2 -> 55.0 | Improved |
| select steps by score instead of randomly (ablation) | ISLA step selection · Math average (Qwen2.5-7B) | average accuracy: 52.0 -> 55.0 | Improved |
| random step selection instead of all steps (ablation) | Step-DPO Qwen2.5-3B · Math average | average accuracy: 45.5 -> 42.6 | Hurt |
| increase selected steps from 6k to 8k (ablation) | ISLA Qwen2.5-7B · Accuracy vs data size | accuracy | Hurt |
| ISLA step scoring instead of PRM-based selection (Qwen2.5-Math-7B-PRM800K) (ablation) | Qwen2.5-7B · Math average | average accuracy: 54.1 -> 55.0 | Improved |
| ISLA step scoring instead of PRM-based selection (Llama-PRM800K) (ablation) | Qwen2.5-7B · Math average | average accuracy: 53.8 -> 55.0 | Improved |
| ISLA vs solution-level DPO | Qwen2.5-7B · GSM8K | accuracy: 87.3 -> 87.3 | No clear effect |
Closest papers by meaning
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.90)
- Inference-Time Reward Hacking in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.89)
- Know What You Don't Know: Uncertainty Calibration of Process Reward Models (NeurIPS 2025 · similarity 0.89)
- Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.89)
- Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
