Research map / Large language models / Instruction and fine-tuning
TS$^2$: Training with Sparsemax+, Testing with Softmax for Accurate and Diverse LLM Fine-Tuning
XuZiyang, Ananthu Rajendran Pillai, Yinghua Yao, Yuangang Pan
This paper tried a common change: Add label smoothing
add label smoothing (0.2) to CE · AlpacaEval BoN@32 · win rate (%): 29.77 -> 28.25 Hurt
What 190 accepted papers report for this change
- 127 improved
- 19 no clear effect
- 7 mixed
- 37 hurt
- Anti-Backdoor Coreset Selection via Cumulative Entropy ICML 2026 · add unlearning step with label smoothing · CIFAR10 WaNet · poisoning rate of coresetImproved
- When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models ICML 2026 · add label smoothing to BT loss · RewardBench Reasoning · accuracy: 75.41 -> 72.28Hurt
- Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision–Language Models ICML 2026 · apply label smoothing instead of plain CE · Average of 9 datasets, 40% sym noise · accuracy (%): 78.09 -> 78.39Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with Sparsemax+ and decode with softmax instead of cross-entropy | Llama-3.1-8B SFT · AlpacaEval BoN@32 | win rate (%): 29.77 -> 33.12 | Improved |
| train with Sparsemax+ and decode with softmax instead of cross-entropy | Qwen-2-7B SFT · AlpacaEval BoN@32 | win rate (%): 31.41 -> 37.48 | Improved |
| use TS2 instead of GEM | Llama-3.1-8B SFT · AlpacaEval BoN@32 | win rate (%): 31.53 -> 33.12 | Improved |
| use TS2 instead of CE | Llama-3.1-8B SFT · HumanEval pass@100 | pass rate (%): 72.60 -> 87.00 | Improved |
| use TS2 instead of CE | Qwen-2-7B SFT · HumanEval pass@100 | pass rate (%): 86.60 -> 91.50 | Improved |
| use TS2 instead of GEM | Qwen-2-7B SFT · HumanEval pass@1 | pass rate (%): 41.80 -> 42.20 (CE 44.90) | Hurt |
| use TS2 instead of GEM | Llama-3.1-8B SFT · OpenLLM six-task average BoN@32 | accuracy (%): 75.69 -> 88.88 | Improved |
| use TS2 instead of GEM for diversity | Llama-3.1-8B SFT · AlpacaEval N-gram diversity | diversity: 20.32 -> 23.78 | Improved |
| add suppression term to sparsemax decoupled training (full TS2 vs Decoupling Only) (ablation) | TS2 · AlpacaEval | win rate | Improved |
| decode with softmax instead of sparsemax at inference (vs Unified Sparsemax) (ablation) | TS2 · AlpacaEval | diversity | Improved |
| add sparsemax support set to suppression penalty (vs Suppression Only on CE) (ablation) | CE with suppression · AlpacaEval | win rate and diversity | Improved |
| increase suppression weight alpha from 0.25 to 5.00 (ablation) | TS2 · AlpacaEval BoN@32 | win rate (%): 33.12 -> 25.34 | Hurt |
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.91)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.90)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.90)
- SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models (ICML 2024 · similarity 0.90)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.90)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
