Research map / Architectures and efficiency / Pruning and efficient tuning
Decoupled Multi-Predictor Optimization for Inference-Efficient Model Tuning
Liwei Luo, Shuaitengyuan Li, Dongwei Ren, Qilong Wang, Pengfei Zhu, Qinghua Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DMPO (bypass modules, high-order predictors, two-phase loss weights) instead of Dyn-Adapter at 30% FLOPs | early-exit PEFT on ViT-B/16 · CIFAR-100 + 5 FGVC (average) | average accuracy: 81.67 -> 88.08 | Improved |
| add multi-predictor architecture (bypass + high-order predictor) (ablation) | DMPO · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 81.67 -> 85.16 | Improved |
| add decoupled optimization (two-phase loss weights) (ablation) | DMPO · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 81.67 -> 84.41 | Improved |
| combine MPA and DO (ablation) | DMPO · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 85.16 -> 88.08 | Improved |
| add MPA or DO at 70% FLOPs (ablation) | DMPO · CIFAR-100 + 5 FGVC, 70% FLOPs | average accuracy: 86.80 -> 86.98 (MPA), 86.96 (DO) | No clear effect |
| use R2D order (small early weights first, large later) instead of D2R (ablation) | decoupled optimization · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 86.43 -> 88.08 | Improved |
| use constant loss weights (early<deep or early>deep) instead of two-phase R2D (ablation) | decoupled optimization · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 88.08 -> 85.03 and 83.86 | Hurt |
| add Sigmoid(loss) inter-stage control factor (ablation) | DMPO · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 87.85 -> 88.08 | Improved |
| use Softmax or Tanh in place of Sigmoid for the control factor (ablation) | DMPO · CIFAR-100 + 5 FGVC, 30% FLOPs | average accuracy: 88.08 -> 80.69 (Tanh), 74.57 (Softmax) | Hurt |
| DMPO at 30% FLOPs versus token pruning DyT at 70% FLOPs on VTAB-1K | DMPO · VTAB-1K | average accuracy: 76.5 (DyT 0.7) vs 76.0 (DMPO 0.3) | Mixed |
| DMPO instead of DyT on ImageNet-Sketch at 70% FLOPs | DMPO · ImageNet-Sketch domain generalization | accuracy: 25.7 -> 21.5 | Hurt |
Closest papers by meaning
- LEAD: Exploring Logit Space Evolution for Model Selection (CVPR 2024 · similarity 0.86)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.84)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.84)
- Faster Parameter-Efficient Tuning with Token Redundancy Reduction (CVPR 2025 · similarity 0.84)
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.84)
- Understanding and Improving Optimization in Predictive Coding Networks (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
