Research map / Large language models / Agents, code and math
Generalizing Verifiable Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IFBench, a held-out benchmark of 58 new verifiable output constraints, and IF-RLVR, GRPO-based RL with verifiable rewards on combined constraints, showing large out-of-domain instruction-following gains.
Problem
Precise instruction following with verifiable output constraints: models overfit to the small IFEval constraint set and fail to generalize to unseen constraint types.
Approach
Build a benchmark with unseen, human-written verifiable constraints (IFBench) separate from training constraints (IFTrain), and train with GRPO using verifiable rewards on multi-constraint prompts with wider variable ranges, plus optional reward-model mixing to mitigate reward hacking.
Key results
- IF-RLVR on Llama-3.1-Tulu-3-8B-DPO raises IFEval from 82.4 to 92.2 and IFBench from 28.9 to 45.9 (abstract/intro; Table reports 44.6 for DPO base).
- IF-RLVR on Qwen2.5-7B reaches 87.8 on IFEval and 53.7 on IFBench, while leading models such as GPT-4.1 or Claude 3.7 Sonnet score below 50% on IFBench.
- Training on 1-6 constraints per instance improves out-of-domain IFBench over single-constraint training (e.g., 59.5 vs 48.9 with up to 3 constraints for Qwen2.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Tulu-DPO with IF-RLVR | Tulu-DPO · IFEval | accuracy: 81.1 -> 92.2 | Helped |
| Train Tulu-DPO with IF-RLVR | Tulu-DPO · IFBench | accuracy: 25.5 -> 44.6 | Helped |
| Train with GRPO and IF verifiable rewards instead of DPO | Tulu-3-8B-SFT · IFEval | accuracy: 76.89 -> 85.77 | Helped |
| Train with GRPO and IF verifiable rewards instead of DPO | Tulu-3-8B-SFT · IFBench | accuracy: 25.2 -> 28.6 | Helped |
| Train with GRPO and IF verifiable rewards instead of DPO | Tulu-3-8B-DPO · IFEval | accuracy: 79.67 -> 89.65 | Helped |
| Train with GRPO and IF verifiable rewards instead of DPO | Tulu-3-8B-DPO · IFBench | accuracy: 29.3 -> 30.6 | Helped |
| Train on more constraints per prompt (ablation) | Qwen2.5 policy · IFBench | accuracy: 48.9 (1 constraint); 59.5 (3 constraints); 55.8 (5 constraints) | Mixed |
| Train on more constraints per prompt (ablation) | Qwen2.5 policy · IFEval | accuracy: 71.2 (1 constraint); 85.8 (6 constraints) | Helped |
| Use a wider training range for constraint variables rather than a disjoint range (ablation) | IF-RLVR · IFEval | accuracy | Helped |
| Use a disjoint training range for constraint variables (ablation) | IF-RLVR · IFEval | accuracy | Hurt |
| Remove length and keyword constraints from training (ablation) | IF-RLVR · IFEval | accuracy | Hurt |
| Remove change-case and detectable-format constraints from training (ablation) | IF-RLVR · IFEval | accuracy: 89.65 | No clear effect |
Surprising findings
- RLVR-trained models score lower than base policies on LLM-as-judge general quality (6.4 vs 7 out of 10), indicating reward hacking toward constraint-following.
- Qwen2.5-72B-Instruct's IF accuracy is most negatively correlated with LLM-as-judge scores among the analyzed models.
Limitations
- Only verifiable constraints are studied, which may not reflect real user constraints.
- RLVR-tuned models slightly degrade on other capabilities such as AlpacaEval 2 and can over-prioritize constraints over task quality.
Experimental setup
Datasets: Tulu-3-SFT WildChat IFEval constraints IFTrain
Benchmarks: IFBench IFEval AlpacaEval 2 GSM8K MMLU BBH
Metrics: prompt-level loose accuracy strict accuracy LLM-as-judge score
Baselines: IFEval-trained models DPO Tulu-3-8B-DPO GPT-4.1 Claude 3.7 Sonnet Qwen3-32B Claude 4 Sonnet o3 DeepSeek R1
Builds on: GRPO IFEval RLVR DPO Tulu 3
Closest papers by meaning
- IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.91)
- Do LLMs ``know'' internally when they follow instructions? (ICLR 2025 · similarity 0.90)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.90)
- Checklists Are Better Than Reward Models For Aligning Language Models (NeurIPS 2025 · similarity 0.89)
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data (NeurIPS 2025 · similarity 0.89)
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.