Research map / Large language models / Instruction and fine-tuning
NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NICE, a policy-gradient-based influence estimation method that selects instruction-tuning data to directly improve non-differentiable evaluation metrics (AlpacaEval win rate, reward model scores, pass@k), including label-free selection.
Problem
Data selection for instruction tuning typically uses NTP loss as a proxy for task performance, which misaligns with non-differentiable long-form generation metrics such as code pass rate, LLM-judge win rate, and reward model scores; label-dependent validation also limits applicability.
Approach
Treat the evaluation metric as a reward and compute the Monte-Carlo policy gradient on validation prompts to estimate each training point's influence on the metric (via TracIn or influence functions), so no labels are needed when the metric is label-free; optionally use a stronger model (GPT-4) to generate assisted samples.
Key results
- On Llama2-7B task-agnostic pool, NICE reaches 26.94 AlpacaEval win rate vs 17.40 for LESS and 22.59 for full data; NICEAMC reaches 30.45.
- On Mistral-7B task-agnostic AlpacaEval, NICEAMC gets 47.40 vs 35.84 for LESS and 33.77 for full dataset.
- NICE-selected subsets outperform loss-based LESS across models, settings, and datasets, and label-free NICE beats label-using baselines on TLDR, RLHF, and HumanEval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NICE policy-gradient influence selection instead of LESS loss-based influence selection | Llama2-7B and Mistral-7B · AlpacaEval, TLDR, RLHF, and HumanEval (task-agnostic pool) | task-specific evaluation metric: AlpacaEval: 26.94±2.37 -> 27.61±2.13 (Llama2-7B); 41.09±1.56 -> 41.43±3.00 (Mistral-7B). TLDR: 3.37±0.29 -> 3.61±0.78; 4 | Helped |
| Use NICEAMC instead of NICE (ablation) | Llama2-7B and Mistral-7B · Task-agnostic pool: AlpacaEval, TLDR, RLHF, and HumanEval | task-specific evaluation metric: Llama2-7B: 27.61±2.13 -> 30.45±2.40; 3.61±0.78 -> 3.55±0.40; 2.82±0.10 -> 3.03±0.02; 48.59±2.08 -> 45.10±2.84. Mistral-7 | Mixed |
| Use NICE instead of training on the full dataset | Llama2-7B and Mistral-7B · AlpacaEval, TLDR, RLHF, and HumanEval (task-agnostic pool) | task-specific evaluation metric: AlpacaEval: 22.59 -> 27.61±2.13 (Llama2-7B); 33.77 -> 41.43±3.00 (Mistral-7B). TLDR: 2.40 -> 3.61±0.78; 2.79 -> 4.80±0.1 | Helped |
| Use NICE instead of LESS loss-based influence selection | Llama2-7B · AlpacaEval, TLDR, RLHF, and HumanEval (task-aware pool) | task-specific evaluation metric: RLHF: 1.62±0.05 -> 1.69±0.05; HumanEval: 52.67±0.71 -> 55.09±1.66 | Helped |
| Use NICE instead of LESS loss-based influence selection | Mistral-7B · RLHF and HumanEval (task-aware pool) | task-specific evaluation metric: RLHF: 1.29±0.13 -> 1.71±0.01; HumanEval: 85.26±1.13 -> 87.35±1.03 | Helped |
| Apply policy-gradient influence estimation to DataInf / Influence Function (ablation) | Llama2-7B · AlpacaEval, TLDR, RLHF, and HumanEval (task-agnostic pool) | task-specific evaluation metric: AlpacaEval: 11.11 -> 20.44; TLDR: 2.01 -> 3.97; RLHF: 0.83 -> 1.89; HumanEval: 37.40 -> 39.68 | Helped |
| Increase the number of Monte-Carlo samples used to estimate the policy gradient (ablation) | NICE on RLHF · RLHF | reward model score | Helped |
| Increase the selected-data percentage in the task-aware pool (ablation) | NICE on RLHF · RLHF | reward model score | Mixed |
| Increase the selected-data percentage in the task-agnostic pool (ablation) | NICE on RLHF · RLHF | reward model score | Hurt |
| Expand the NICEAMC selection pool from the RLHF set to the RLHF set plus a large instruction-tuning set (ablation) | Llama2-7B · RLHF | reward model score: 1.26 -> 3.35 | Helped |
| Expand the NICE selection pool from the RLHF set to the RLHF set plus a large instruction-tuning set (ablation) | Llama2-7B · RLHF | reward model score: 1.68 -> 2.44 | Helped |
Surprising findings
- Minimized validation NTP loss checkpoint was the worst-performing checkpoint by reward model score on RLHF; reward kept rising after loss increased.
- NICEAMC does not always beat NICE; with a small selection pool (task-aware), NICE can outperform NICEAMC.
Limitations
- MC policy gradient is computationally expensive, especially for large validation sets or many samples (e.g., 500 samples for HumanEval).
- When the policy is too weak, MC samples may lack high-reward responses, weakening the signal; AMC relies on a stronger model like GPT-4.
- Performance on task-aware small pools can drop sharply when selection percentage grows beyond 25%.
Experimental setup
Datasets: Tulu AlpacaEval TLDR RLHF (Anthropic HH helpfulness) HumanEval CodeAlpaca 20k
Benchmarks: AlpacaEval TLDR RLHF HumanEval
Metrics: length-controlled win rate reward model score pass@100
Baselines: Random RDS BM25 DSIR TSDS LESS DataInf
Builds on: LESS TracIn Influence Function DataInf REINFORCE
Closest papers by meaning
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.90)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.90)
- T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning (NeurIPS 2025 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.89)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.