Research map / Large language models / Instruction and fine-tuning
Explainable Token-level Noise Filtering for LLM Fine-tuning Datasets
Yuchen Yang, Wenze Lin, Enhao Huang, Zhixuan Chu, Hongbin zhou, Lan Tao, Yiming Li, Zhan Qin, Kui Ren
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add XTF token-level gradient masking to fine-tuning | LLM fine-tuning · GSM8K (average of 10 cases) | accuracy: 37.1 -> 45.7 | Improved |
| add XTF token-level gradient masking to fine-tuning | LLM fine-tuning · PubMedQA (average of 10 cases) | accuracy: 30.4 -> 37.1 | Improved |
| use XTF instead of sample-level data filtering | Deepseek-distilled-qwen-1.5B · GSM8K | accuracy: 47.0 -> 56.2 | Improved |
| use XTF instead of selective language model training | Llama-3.1-8B full fine-tuning · GSM8K | accuracy: 60.3 -> 58.7 | Hurt |
| train for twice as many epochs | Normal fine-tuning · PubMedQA (average) | accuracy: 30.4 -> 29.9 | No clear effect |
| add data augmentation | Normal fine-tuning · GSM8K (average) | accuracy: 37.1 -> 36.5 | No clear effect |
| add all three attribute filters (RI, KN, TR) instead of none (ablation) | XTF · GSM8K / PubMedQA (average of 6) | accuracy: 30.7 -> 40.1 | Improved |
| add KN filtering to RI+TR (ablation) | XTF · GSM8K / PubMedQA (average of 6) | accuracy: 36.9 -> 40.1 | Improved |
| add TR filtering to RI+KN (ablation) | XTF · GSM8K / PubMedQA (average of 6) | accuracy: 36.1 -> 40.1 | Improved |
| add RI filtering to KN+TR (ablation) | XTF · GSM8K / PubMedQA (average of 6) | accuracy: 36.3 -> 40.1 | Improved |
| filter with KN alone versus RI alone (ablation) | XTF single attribute · GSM8K / PubMedQA (average of 6) | accuracy: 32.0 -> 34.5 | Improved |
| apply XTF to larger base models rather than smaller | XTF on code · HumanEval | pass@k | Improved |
Closest papers by meaning
- Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning (ICML 2025 · similarity 0.92)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.90)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.89)
- Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning (NeurIPS 2025 · similarity 0.89)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
