Research map / Large language models / Alignment and preferences
R.I.P.: Better Models by Survival of the Fittest Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RIP, which filters preference-training prompts by the quality of their rejected responses and the reward gap between chosen and rejected responses, and Self-RIP, which uses RIP-curated prompts as few-shot seeds to generate higher-quality synthetic prompts for DPO.
Problem
Training data quality matters for post-training, but existing prompt-curation heuristics ignore the model responses; the paper asks whether response-level statistics of preference pairs can identify low-quality prompts for DPO.
Approach
Low-quality prompts yield low-quality, high-variance responses, so filter prompts using rejected-response reward/length and the chosen-rejected reward gap; reuse the filtered prompts as few-shot seeds for synthetic prompt generation (Self-RIP) with a second RIP filtering pass.
Key results
- Llama 3.1-8B-Instruct DPO on WildChat with RIP filtering raises AlpacaEval2 LC win rate from 48.4 (no filtering) to 57.8, using 4538 of 20k prompts.
- Llama 3.3-70B-Instruct + RIP improves Arena-Hard from 70.5 (no filtering) to 82.9 and AlpacaEval2 LC win rate from 54.3 to 67.7.
- Self-RIP synthetic prompts reach 60.2 AlpacaEval2 LC win rate, versus 49.1 for Self-Instruct data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter WildChat prompts using rejected-response reward, rejected-response length, and reward gap | Llama 3.1-8B-Instruct trained on WildChat-20k · AlpacaEval2 | LC Win: 48.4 -> 57.8 | Helped |
| Filter WildChat prompts using rejected-response reward, rejected-response length, and reward gap | Llama 3.1-8B-Instruct trained on WildChat-20k · Arena-Hard | Score: 37.9 -> 43.1 | Helped |
| Filter WildChat prompts using rejected-response reward, rejected-response length, and reward gap | Llama 3.1-8B-Instruct trained on WildChat-20k · WildBench | WB-Score: 41.5 -> 45.6 | Helped |
| Filter WildChat-40k prompts using RIP | Llama 3.3-70B-Instruct · Arena-Hard | Score: 70.5 to 82.9 | Helped |
| Use Llama 3.1-8B-Instruct rather than Llama 3.3-70B-Instruct to filter (ablation) | Llama 3.3-70B-Instruct trained on WildChat · Arena-Hard | Score: 70.5 -> 76.7; 82.9 | Mixed |
| Use RIP-curated few-shot prompts to generate synthetic instructions and apply RIP post-filtering | Llama 3.1-8B-Instruct trained on synthetic prompts · AlpacaEval2 | LC Win: 49.1% to 60.2% | Helped |
| Use RIP-curated few-shot prompts to generate synthetic instructions and apply RIP post-filtering | Llama 3.1-8B-Instruct trained on synthetic prompts · Arena-Hard | Score: 38.5% to 42.1% | Helped |
| Remove RIP post-filtering of Self-RIP generated prompts (ablation) | Llama 3.1-8B-Instruct trained on Self-RIP prompts · Arena-Hard | Score: 43.7 -> 42.1 | Hurt |
| Use unfiltered Self-Instruct few-shot examples instead of RIP-selected examples (ablation) | Llama 3.1-8B-Instruct trained on generated prompts with RIP post-filtering · Arena-Hard | Score: 43.7 -> 40.9 | Hurt |
| Increase sampled responses from N=8 to N=64 before applying RIP (ablation) | Llama 3.1-8B-Instruct trained on WildChat · Armo Score on valid set | Armo Score: 0.1860 -> 0.1898 | Helped |
| Reduce RIP-curated few-shot seed prompts from 4538 to 256 (ablation) | Llama 3.1-8B-Instruct trained on Self-RIP prompts · AlpacaEval2 | LC Win: 60.2 -> 58.9 | Hurt |
| Apply RIP filtering to prompts generated from unfiltered Self-Instruct examples (ablation) | Llama 3.1-8B-Instruct trained on synthetic prompts · WildBench | WB-Score: 40.0 -> 44.1 | Mixed |
Surprising findings
- Filtering by rejected-response quality and reward gap outperforms SFT-derived filters such as InsTag, IFD, and LLM-as-prompt-judge in the preference setting.
- Using the 8B model to filter for the 70B model gives most of the gain (LC win 64.5 vs 67.7 with 70B filtering).
Limitations
- Gains on Arena-Hard and WildBench for the 8B model are smaller than on AlpacaEval2; RIP scores lower than no filtering on WildBench for Self-RIP (42.5 vs 44.8 without post-filtering).
- Scaling WildChat training beyond 20k prompts saturates, so gains from more data are limited.
- Safety is not explicitly constrained by the reward; the paper leaves safety filtering to future work.
Experimental setup
Datasets: WildChat HelpSteer2 Humpback validation set Evol-Instruct test set
Benchmarks: AlpacaEval2 Arena-Hard WildBench
Metrics: LC win rate win rate Arena-Hard score WB-Score ArmoRM validation score
Baselines: WildChat-20k DPO (no filtering) LLM-as-Prompt-Judge InsTag-Difficulty InsTag-Diversity IFD Perplexity filtering Jaccard Similarity Self-Instruct GPT-4 Omni (05/13) GPT-4 Turbo (04/09)
Builds on: DPO Self-Instruct ArmoRM IFD INSTRUCTMINING
Closest papers by meaning
- AlpaGasus: Training a Better Alpaca with Fewer Data (ICLR 2024 · similarity 0.85)
- Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates (ICLR 2025 · similarity 0.84)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.84)
- T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning (NeurIPS 2025 · similarity 0.83)
- NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric (ICML 2025 · similarity 0.83)
- Improving Data Efficiency via Curating LLM-Driven Rating Systems (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.