Research map / Architectures and efficiency / Attention and transformers
Sparsity Forcing: Reinforcing Token Sparsity of MLLMs
Feng Chen, Yefei He, Lequan Lin, Jing Liu, Chenhui Gou, Bohan Zhuang, Qi Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-train with Sparsity Forcing (RL) on top of ZipVL | Qwen2.5-VL-7B · 7 image benchmarks avg | Avg. score: ratio 79.5% -> 24.7%; avg 72.9 -> 73.6 | Improved |
| Sparsity Forcing vs full attention | Qwen2.5-VL-7B · 7 image benchmarks avg | Avg. score: 73.8 -> 73.6 at 24.7% tokens | No clear effect |
| Sparsity Forcing instead of FastV/VisionZip | Qwen2.5-VL-3B · 7 image benchmarks avg | Avg. score: 65.1/65.8 -> 68.7 | Improved |
| Sparsity Forcing vs full attention on video | LLaVA-Video-7B · 6 video benchmarks avg | Avg. score: 61.4 -> 61.3 at 29.6% tokens | No clear effect |
| RL with joint reward vs SFT sparsity baselines (MOBA, sharpness loss) | Qwen2.5-VL-7B · Post-training comparison avg | Avg. score: 66.2 / 66.8 -> 70.0 | Improved |
| top-p sparse attention instead of top-k (ablation) | Sparsity Forcing · VideoMME | VideoMME: 60.2 -> 64.0 | Improved |
| top-p instead of threshold pruning (ablation) | Sparsity Forcing · VideoMME | VideoMME: 61.6 -> 64.0 | Improved |
| lower training p range [0.80,0.95] (ablation) | Sparsity Forcing · VideoMME | VideoMME: 64.0 -> 63.6 | No clear effect |
| group size 12 instead of 8 (ablation) | Sparsity Forcing · MME/VideoMME | VideoMME: 64.0 -> 63.9 | No clear effect |
| add latency to efficiency reward (ablation) | Sparsity Forcing · 6 video benchmarks avg | Avg. score: 61.3 -> 61.2 | No clear effect |
| replace sharpness-loss with RL-based Sparsity Forcing over temperature adjustment | Qwen2.5-VL-7B · Post-training comparison avg | Avg. score: 67.4 -> 70.0 | Improved |
Closest papers by meaning
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.91)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.90)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.90)
- Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.90)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.90)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
