Research map / Large language models / Instruction and fine-tuning
ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning
Xiaohan Qin, Xiaoxing Wang, Ning Liao, Cancheng Zhang, Xiangdong Zhang, Mingquan Feng, Jingzhi Wang, Junchi Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention-based semantic score to loss-based REL selection (gamma 1.0 -> 0.5) (ablation) | ssToken · 10-benchmark average | average score: LLaMA-3.1-8B 58.61 -> 59.44 | Improved |
| add REL loss signal to attention-only selection (gamma 0.0 -> 0.5) (ablation) | ssToken · 10-benchmark average | average score: LLaMA-3.1-8B 57.80 -> 59.44 | Improved |
| combine signals (gamma 0.5) versus attention-only (gamma 0.0) (ablation) | ssToken · Qwen-2.5-14B 10-benchmark average | average score: 65.62 -> 65.84 | Improved |
| use deeper layer attention for semantic score (ablation) | ssToken · LLaMA-3.2-3B 10-benchmark average | average score: 51.63 -> 52.50 (shallow -> deep) | Improved |
| select 60% of tokens instead of 20% on LLaMA-3.2-3B (ablation) | ssToken · LLaMA-3.2-3B 10-benchmark average | average score: 51.72 -> 52.50 | Improved |
| raise selection ratio from 0.6 to 0.8 (ablation) | ssToken · Qwen-2.5-14B 10-benchmark average | average score: 65.16 -> 65.84 | Improved |
| enable EMA update of the history model (ablation) | ssToken · SFT data pool | average score | No clear effect |
| use ssToken instead of full-data fine-tuning | four base models · 10-benchmark average | average score: LLaMA-3.2-3B 50.35 -> 52.50; Qwen-2.5-7B 59.73 -> 60.48 | Improved |
| use RHO-1 or TokenCleaning (reference-model token selection) instead of full-data | Qwen-2.5-7B · 10-benchmark average | average score: 59.73 -> 58.49 (RHO-1) | Hurt |
| use ssToken instead of TokenCleaning | four base models · 10-benchmark average | average score: Qwen-2.5-7B 58.83 -> 60.48; LLaMA-3.1-8B 59.14 -> 59.44 | Improved |
| random token selection instead of full data | LLaMA-3.2-3B · 10-benchmark average | average score: 50.35 -> 49.82 | No clear effect |
Closest papers by meaning
- Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning (ICML 2025 · similarity 0.91)
- OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference (ICML 2025 · similarity 0.90)
- Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing (ICML 2025 · similarity 0.88)
- Logits are All We Need to Adapt Closed Models (ICML 2025 · similarity 0.88)
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models (NeurIPS 2024 · similarity 0.88)
- Unlocking Full Efficiency of Token Filtering in Large Language Model Training (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
