Research map / Vision-language and multimodal / Multimodal LLMs
p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
Jun Zhang, Desen Meng, Zhengming Zhang, Zhenpeng Huang, Tao Wu, Limin Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TanhNorm to vanilla MoD (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 60.4 -> 61.4 | Improved |
| add STRing symmetric token reweighting (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 61.4 -> 63.4 | Improved |
| add progressive ratio decay (PRD) (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 63.4 -> 65.3 | Improved |
| use TanhNorm instead of softmax weight normalization (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 62.9 -> 65.3 | Improved |
| use TanhNorm instead of shifted softmax (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 62.4 -> 65.3 | Improved |
| use gating factor alpha 0.2 instead of 1 in TanhNorm (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | stability: overflow -> 65.3 | Improved |
| add weight normalization (TanhNorm) over none (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | stability: overflow -> 65.3 | Improved |
| use decaying ratio schedules (cosine, linear) over constant/interleaved (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG | Improved |
| use shifted cosine decay instead of stepped decay (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG | Improved |
| raise shift factor beta from 0.3 to 0.5 (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | AVG: 63.4 -> 65.3 | Improved |
| raise shift factor beta from 0.3 to 0.5 (ablation) | p-MoD · AVG of 6 benchmarks (ablation) | TFLOPs: 17.78 -> 21.94 | Hurt |
| apply p-MoD to LLaVA-NeXT-7B | p-MoD · AVG over 15 benchmarks | TFLOPs: 39.46 -> 21.94 | Improved |
Closest papers by meaning
- $\gamma-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models (ICLR 2025 · similarity 0.91)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.90)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.90)
- DenseMLLM: Standard Multimodal LLMs for Dense Prediction (ICML 2026 · similarity 0.90)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.90)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
