Research map / Large language models / Instruction and fine-tuning
Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes token cleaning, which scores each SFT response token by the loss change between a base and reference model and keeps only the top ~60% of tokens, either with a fixed reference or a self-evolving reference updated over data splits.
Problem
Standard SFT treats every response token as a valid target, so non-task-related or uninformative tokens within otherwise high-quality samples can dilute the training signal and hurt downstream performance.
Approach
Treat token-level labels as noisy and score tokens by the influence of a model update (negative loss change between a base model and a reference model), then keep the top-k% tokens globally; the reference model can be fixed (one-shot) or iteratively updated on successive data splits (self-evolving).
Key results
- Self-evolving cleaning reaches 53.00 average vs 49.87 for full tokens on LLaMA-3.2-3B (about 6.3% relative gain as stated), with the best token proportion around 0.6.
- On LLaMA-3.1-8B, self-evolving cleaning averages 59.20 vs 56.70 for full tokens and 58.55 for Rho.
- Global token ranking (Fixed-Model Cleaning, 51.98) outperforms local per-sample ranking Rho (51.46) on LLaMA-3.2-3B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use token cleaning instead of full-token SFT | LLaMA-3.2-3B · Seven-benchmark average | AVG: 49.87 -> 53.00 | Helped |
| Use token cleaning instead of full-token SFT | LLaMA-3.1-8B · Seven-benchmark average | AVG: 56.70 -> 59.20 | Helped |
| Use token cleaning instead of full-token SFT | Mistral-7B-v0.3 · Seven-benchmark average | AVG: 54.12 -> 55.20 | Helped |
| Replace local per-sample token ranking with global dataset-level ranking | Fixed-Model Cleaning · Seven-benchmark average, LLaMA-3.2-3B | AVG: 51.46 -> 51.98 | Helped |
| Replace local per-sample token ranking with global dataset-level ranking | Fixed-Model Cleaning · Seven-benchmark average, Mistral-7B-v0.3 | AVG: 53.60 -> 55.20 | Helped |
| Iteratively update the reference model instead of keeping it fixed | Token cleaning · Seven-benchmark average, LLaMA-3.2-3B | AVG: 51.98 -> 53.00 | Helped |
| Iteratively update the reference model instead of keeping it fixed | Token cleaning · Seven-benchmark average, LLaMA-3.1-8B | AVG: 58.90 -> 59.20 | Helped |
| Iteratively update the reference model instead of keeping it fixed | Token cleaning · Seven-benchmark average, Mistral-7B-v0.3 | AVG: 55.20 -> 55.00 | Hurt |
| Select 0.6 of tokens rather than all tokens (ablation) | Fixed-Model Cleaning on LLaMA-3.2-3B · Seven-benchmark average | AVG: 49.87 -> 51.98 | Helped |
| Select 0.6 of tokens rather than all tokens (ablation) | Self-Evolving Cleaning on LLaMA-3.2-3B · Seven-benchmark average | AVG: 49.87 -> 53.0 | Helped |
| Increase selected-token proportion from 0.6 to 0.8 (ablation) | Self-Evolving Cleaning on LLaMA-3.2-3B · Seven-benchmark average | AVG: 53.0 -> 49.25 | Hurt |
| Use LLaMA-3.1-8B-Instruct rather than the warmup model as reference (ablation) | Fixed-Model Cleaning · TydiQA | 1-shot F1: 52.60 -> 39.17 | Hurt |
Surprising findings
- Performance gains largely rely on a small fraction of highly informative tokens; selecting 100% of tokens is consistently worse than 50-70%.
- A stronger reference model (LLaMA-3.1-8B-Instruct) did not improve MMLU or BoolQ under token cleaning, suggesting distribution shift.
Limitations
- Experiments limited to 3B-8B base models.
- Data pool limited to 50k samples due to cost.
- Token proportion is a fixed hyperparameter; no method to estimate token noise ratio.
Experimental setup
Datasets: Flan_v2 Open Assistant 1 Stanford Alpaca Dolly WizardLM
Benchmarks: MMLU TruthfulQA TydiQA HellaSwag ARC-Challenge BoolQ LogiQA
Metrics: accuracy MC2 1-shot F1
Baselines: Base DS2 Full Tokens Uniform Random Rho
Builds on: RHO-1 (selective language modeling) DS2 (data selection via self-correction) LoRA
Closest papers by meaning
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning (NeurIPS 2025 · similarity 0.89)
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.89)
- Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing (ICML 2025 · similarity 0.88)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.88)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.