Research map / Large language models / Instruction and fine-tuning
Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DaaR, which uses an LLM's frozen embeddings and an MLP probe to predict predictive-entropy-based diversity rewards, selecting the top 20% of domain-unlabeled data for LoRA fine-tuning.
Problem
Fine-tuning LLMs on diverse multi-domain data when domain labels are missing, imprecise, or non-normalized; existing mixture methods need domain labels and selection methods struggle to balance multi-domain performance.
Approach
Model-aware centroid synthesis from the LLM's own generated seed data produces pseudo-labels; a frozen-LLM MLP domain probe is trained, and a second MLP regresses its predictive entropy as a label-free diversity reward used to select the top 20% of samples.
Key results
- On Qwen2.5-7B, DaaR reaches 53.76 Avg versus 52.66 for the best baseline (Deita-Best 52.47 and Random 52.65 nearby), and 53.99 vs 53.02 Random on Qwen2-7B.
- On Llama3.1-8B, DaaR achieves 38.83 Avg, +0.14 over the best baseline SuperFilter (38.73) and +2.97 over Random (35.86 raw; 37.27 random).
- DaaR reports 38.1 on MATH for Qwen2 versus 35.36 for Random, and claims the probe reaches 92.7% domain validation accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select data by predicted entropy reward rather than the strongest average baseline | DaaR fine-tuning · 7-task Avg on Llama3.1-8B | Avg: 38.73 -> 38.83 (+0.14) | Helped |
| Select data by predicted entropy reward rather than random sampling | DaaR fine-tuning · 7-task Avg on Qwen2-7B | Avg: 53.02 -> 53.99 (+0.97) | Helped |
| Select data by predicted entropy reward rather than random sampling | DaaR fine-tuning · 7-task Avg on Qwen2.5-7B | Avg: 52.65 -> 53.76 (+1.11) | Helped |
| Select data by predicted entropy reward rather than random sampling | DaaR fine-tuning · MMLU-Avg on Qwen2-7B | accuracy: 68.81 -> 69.42 (0.61 points) | Helped |
| Select data by predicted entropy reward rather than the strongest baseline | DaaR fine-tuning · Avg on Qwen3-8B | Avg: 49.90 -> 50.06 | Helped |
| Use model-aware pseudo-labels instead of ground-truth labels (ablation) | DaaR on Llama3.1-8B · 7-task Avg | Avg: 38.65 -> 38.83 (+0.18) | Helped |
| Use model-aware pseudo-labels instead of ground-truth labels (ablation) | DaaR on Qwen2-7B · 7-task Avg | Avg: 53.40 -> 53.99 (+0.59) | Helped |
| Use model-aware pseudo-labels instead of ground-truth labels (ablation) | DaaR on Qwen2.5-7B · 7-task Avg | Avg: 53.48 -> 53.76 (+0.28) | Helped |
| Remove minor injection of downstream-task samples during seed generation (ablation) | DaaR seed generation · Final performance | final performance | No clear effect |
| Vary the selected hidden layer across layers 1-10 (ablation) | DaaR domain predictor · Domain classification validation | accuracy: 91.2-93.6% | No clear effect |
| Vary the number of seed samples (ablation) | DaaR centroid construction · Centroid stability | centroid similarity variance: 5.4E-7 to 1.8E-5 | No clear effect |
| Vary the number of augmented samples (ablation) | DaaR centroid construction · Centroid stability | mean centroid similarity: 0.983 and 0.981 | No clear effect |
Surprising findings
- Optimal diversity thresholds vary across models and tasks; for Llama3.1-8B the top inter-diversity bucket helps while the bottom one falls below random.
- Most baselines fall below the random selection baseline on Qwen models, indicating distribution bias when selecting for multi-domain mixtures.
Limitations
- Optimal selection ratios and diversity parameters are sensitive to task and model, requiring manual customization.
- Gains over the best baseline are small (e.g., +0.14 on Llama3.1-8B), and results are averaged over only two seeds with a small evaluation subset.
- Uses a fixed 20% selection ratio and LoRA single-epoch training on 8K samples.
Experimental setup
Datasets: Dolly-15K CoT-en Math-Instruct Code-Alpaca NQ TriviaQA HellaSwag GSM8K MATH MBPP HumanEval MMLU
Benchmarks: NQ TriviaQA HellaSwag GSM8K MATH MBPP HumanEval MMLU Qwen3 eval suite
Metrics: accuracy average score (Avg)
Baselines: Random Selection Instruction Length AlpaGasus Instag-Best SuperFilter Deita-Best
Builds on: Deita Instag AlpaGasus SuperFilter
Closest papers by meaning
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- Can DPO Learn Diverse Human Values? A Theoretical Scaling Law (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.