Research map / Architectures and efficiency / Pruning and efficient tuning
Navigating Extremes: Dynamic Sparsity in Large Output Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies semi-structured fixed fan-in dynamic sparse training (SET-style magnitude pruning and random regrowth) to the classifier of end-to-end extreme multi-label classification, with an intermediate layer and a decaying auxiliary cluster-based loss to restore encoder gradient flow.
Problem
Classification layers over millions of labels dominate GPU memory in end-to-end extreme multi-label classification, and naive sparsification with DST hampers convergence because poor gradient flow from the sparse classifier to the trainable text encoder.
Approach
Replace the dense XMC output layer with a fixed fan-in semi-structured sparse layer trained with magnitude-based pruning and random regrowth (SET), and add an intermediate layer plus a decaying auxiliary meta-classifier (label-cluster) loss so the encoder receives useful gradients early in training.
Key results
- On Amazon-3M, Spartex at 83% sparsity reaches P@1 50.2 vs dense 53.4 while training memory drops from 46.3 GiB to 13.5 GiB (3.4x reduction).
- On Amazon-670K at 83% sparsity, the auxiliary loss raises P@1 from 45.6 to 47.1 and at 92% sparsity from 30.7 to 42.3.
- Spartex outperforms the same-capacity Dense BN baseline at matched memory (e.g., Wiki-500K P@1 76.7 vs 73.8 at 83% sparsity).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spartex instead of a dense classifier | XMC models · Amazon-3M | M_tr (GiB): 46.3 -> 13.5 | Helped |
| Use Spartex instead of Dense BN | XMC models · Amazon-670K | P@1: 44.5 -> 47.1 | Helped |
| Add an auxiliary loss (ablation) | Spartex, fan-in 128 (83%) · Amazon-670K | P@1: 45.6 -> 47.1 | Helped |
| Add an auxiliary loss (ablation) | Spartex, fan-in 64 (92%) · Amazon-670K | P@1: 30.7 -> 42.3 | Helped |
| Add an auxiliary loss (ablation) | Spartex, fan-in 32 (96%) · Amazon-670K | P@1: 5.5 -> 38.4 | Helped |
| Add an auxiliary loss (ablation) | Spartex, fan-in 256 (67%) · Amazon-670K | P@1: 47.0 -> 47.6 | Helped |
| Continue auxiliary loss throughout training rather than cutting it off (ablation) | Spartex, fan-in 128 (83%) · Amazon-670K | P@1: 47.1 -> 42.1 | Hurt |
| Continue auxiliary loss throughout training rather than cutting it off (ablation) | Spartex, fan-in 64 (92%) · Amazon-670K | P@1: 42.3 -> 22.7 | Hurt |
| Train end-to-end rather than using fixed embeddings (ablation) | DST XMC model · Wiki-500K | P@1: 73.6 -> 76.7 | Helped |
| Train end-to-end rather than using fixed embeddings (ablation) | DST XMC model · Amazon-670K | P@1: 42.6 -> 47.1 | Helped |
| Increase sparsity from 50% to 96% (ablation) | Spartex · Amazon-670K | P@1: 49.2 -> 38.4 | Hurt |
| Use the auxiliary objective at lower sparsity (≤ 67%) (ablation) | Spartex · Amazon-670K | P@1 | No clear effect |
Surprising findings
- Randomly reassigning labels to meta-classifier clusters barely hurts LightXML's final performance, suggesting the auxiliary branch helps mainly through gradient signal rather than label grouping.
- Keeping the auxiliary loss for the full training run degrades performance below the no-auxiliary baseline, so it must be decayed and cut off.
Limitations
- Does not surpass the dense last-layer baseline on all datasets and sparsity levels.
- Performance degrades as label space grows and at high sparsity (>=90%) without the auxiliary loss.
- Tested only with BERT-scale encoders; larger encoders such as RoBERTa-Large are left to future work.
Experimental setup
Datasets: Wiki10-31K Wiki-500K Amazon-670K Amazon-3M LF-AmazonTitles-131K LF-WikiSeeAlso-320K
Benchmarks: Wiki10-31K Wiki-500K Amazon-670K Amazon-3M LF-AmazonTitles-131K LF-WikiSeeAlso-320K
Metrics: P@1 P@3 P@5 PSP@1 PSP@3 PSP@5 peak GPU memory (GiB)
Baselines: CascadeXML LightXML AttentionXML Dense Dense BN RigL Renee NGAME SiameseXML ECLARE DEXML
Builds on: SET RigL CascadeXML Renee
Closest papers by meaning
- Advancing Dynamic Sparse Training by Exploring Optimization Opportunities (ICML 2024 · similarity 0.90)
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.90)
- Dynamic Sparse Training of Diagonally Sparse Networks (ICML 2025 · similarity 0.89)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.88)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.88)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.