Research map / Optimization / Optimizers
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model
Zirui Liu, Guanchu Wang, Shaochen Zhong, Zhaozhuo Xu, Daochen Zha, Ruixiang Tang, Zhimeng Jiang, Kaixiong Zhou, Vipin Chaudhary, Shuai Xu, Xia Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WTA-CRS@0.3 to full fine-tuning | T5-Base · GLUE average | average score: 85.7 -> 85.4 | No clear effect |
| add WTA-CRS@0.3 to LoRA | T5-3B · GLUE average | average score: 89.5 -> 89.8 | Improved |
| add WTA-CRS@0.3 to LoRA | T5-Base · peak memory (GB) | peak memory: 13.84 -> 6.50 | Improved |
| add WTA-CRS@0.3 to full fine-tuning | T5-Large · peak memory (GB) | peak memory: 45.85 -> 21.58 | Improved |
| lower column-row budget from k=0.3|D| to k=0.1|D| (ablation) | WTA-CRS · T5-Base/Large memory | peak memory (GB): 8.44 -> 7.30 | Improved |
| lower column-row budget from k=0.3|D| to k=0.1|D| (ablation) | WTA-CRS · GLUE average | accuracy: about 1% drop | Hurt |
| use WTA-CRS@0.1 on larger model (ablation) | T5-3B · GLUE average | accuracy: 0.4% drop | No clear effect |
| compare LST with LoRA and WTA-CRS | LST on T5-Base · GLUE average | average score: 85.7 -> 83.8 | Hurt |
| replace WTA-CRS with deterministic top-k selection (ablation) | WTA-CRS · T5-Base SST2/MNLI/QQP | validation accuracy | Hurt |
| replace WTA-CRS with plain CRS sampling (ablation) | WTA-CRS · T5-Base SST2/MNLI/QQP | validation accuracy | Hurt |
| combine WTA-CRS with LoRA for larger batches | T5-3B · maximum batch size | batch size enlargement: 4.8x to 6.4x | Improved |
| use WTA-CRS@0.1 with larger batch size | T5-Large · training throughput | throughput: 1.08x | Improved |
Closest papers by meaning
- MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling (NeurIPS 2025 · similarity 0.87)
- A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.87)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.87)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.87)
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.86)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
