Research map / Large language models / Language, safety and interpretability
Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Annotates the Tulu-3-SFT-Mix and SmolTalk SFT datasets with an extended Magpie pipeline (Llama-3.3-70B judge), then builds a quality- and task-aware curated mixture, TuluTalk (808k samples), that matches or beats both sources with 14-23% fewer samples.
Problem
Open post-training SFT datasets (Tulu, SmolTalk) lack systematic, side-by-side comparison and detailed quality annotations, so it is unclear how sample quality, task type, and turn structure affect downstream performance.
Approach
Extend Magpie to tag multi-turn conversations (full-history prompts, error-tolerant JSON parsing, judge-based instruct reward for multi-turn), analyze both datasets along task, turn structure, input quality and response quality, and curate a smaller mixture that keeps high-quality samples while restoring task diversity.
Key results
- On Llama-3.1-8B, TuluTalk (full, 808k) reaches 51.62 overall average vs 51.38 SmolTalk and 50.32 Tulu, with 14% fewer samples than Tulu and 23% fewer than SmolTalk.
- On SmolLM2-1.7B, TuluTalk reaches 35.89 overall average vs 35.49 SmolTalk and 35.16 Tulu.
- Magpie-based human check: input quality exact-match agreement 91% (Spearman 0.85) and instruct reward agreement 93% (Spearman 0.87) on 100 samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full TuluTalk quality- and task-aware mixture instead of SmolTalk | Llama-3.1-8B SFT · Overall (all benchmarks) | average score: 51.38 -> 51.62 | Helped |
| Use the full TuluTalk quality- and task-aware mixture instead of Tulu | SmolLM2-1.7B SFT · Overall (all benchmarks) | average score: 35.16 -> 35.89 | Helped |
| Apply quality-based curation only to form TuluTalk-80k instead of using Tulu-100k (ablation) | Llama-3.1-8B SFT · GSM8K | score: 65.88% vs. 66.64% | Helped |
| Apply quality-based curation only to form TuluTalk-80k instead of using Tulu-100k (ablation) | Llama-3.1-8B SFT · IF-Eval | score: 66.03 -> 64.38 | Hurt |
| Apply quality-based curation only to form TuluTalk-80k instead of using Tulu-100k (ablation) | Llama-3.1-8B SFT · HumanEval | pass@1: 50.61 -> 48.76 | Hurt |
| Add task-aware samples from underrepresented categories to quality-based curation (ablation) | Llama-3.1-8B SFT · GSM8K | score: 66.64% vs. 69.45% | Helped |
| Add task-aware samples from underrepresented categories to quality-based curation (ablation) | Llama-3.1-8B SFT · HumanEval | pass@1: 51.22% vs. 48.76% | Helped |
| Add task-aware samples from underrepresented categories to quality-based curation (ablation) | SmolLM2-1.7B SFT · IF-Eval | score: 49.19% to 52.5% | Helped |
| Add task-aware samples from underrepresented categories to quality-based curation (ablation) | Llama-3.1-8B SFT · Overall (all benchmarks) | average score: 48.33 -> 48.55 | Helped |
Surprising findings
- Tulu is 95% single-turn and almost entirely lacks multi-turn math samples, while SmolTalk is 70% multi-turn.
- Strict quality-only filtering reduced information-seeking and instruction-following diversity, requiring task-aware re-addition of samples to recover benchmark performance.
Limitations
- Only SFT is studied; preference tuning and RL data quality are left to future work (DPO checked only on Llama).
- Annotation relies on a single LLM judge (Llama-3.3-70B) and reward model, with multi-turn instruct reward using a judge rather than a reward model.
Experimental setup
Datasets: Tulu-3-SFT-Mix SmolTalk Orca-AgentInstruct
Benchmarks: MMLU MMLU-Pro TruthfulQA GPQA ARC-C BBH MuSR HellaSwag WinoGrande IF-Eval GSM8K MATH
Metrics: accuracy pass@1 Open LLM Leaderboard average
Baselines: Tulu-3-SFT-Mix SmolTalk Orca-AgentInstruct
Builds on: Magpie Tulu 3 SmolTalk Orca-AgentInstruct DPO
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.92)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.91)
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.91)
- Correlated Errors in Large Language Models (ICML 2025 · similarity 0.91)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.91)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.