Research map / Architectures and efficiency / Pruning and efficient tuning
FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment
Riccardo Zaccone, Stefanos Laskaridis, Marco Ciccone, Samuel Horváth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train nested submodels (NSL) instead of only the full model (PTS) (ablation) | low-rank elastic model · linear-model theory and MNIST toy | submodel optimality gap: gap zero only for NSL | Improved |
| train all submodels jointly (ASL) instead of nested (ablation) | low-rank elastic model · linear-model theory and MNIST toy | submodel optimality gap: strictly positive gap | Hurt |
| add distillation-based submodel training on top of DataSVD + DP selection (ablation) | FlexRank · GPT-2 eval loss | loss | Improved |
| train each layer independently during distillation instead of end-to-end (ablation) | FlexRank · GPT-2 eval loss | loss | Hurt |
| increase DataSVD calibration samples above 128 (ablation) | FlexRank initialization · GPT-2 eval loss | loss | No clear effect |
| train a single submodel at one budget instead of nested sampling (ablation) | FlexRank · GPT-2 eval loss | loss | Hurt |
| use nested weight-shared submodels instead of independent submodels at equal total budget (ablation) | FlexRank · Llama-3.2-1B | average accuracy: slight | Improved |
| replace ACIP LoRA adapters with nested distillation of shared weights | ACIP · Llama models commonsense average | average accuracy | Improved |
| use DP rank selection over uniform truncation | FlexRank · GPT-2 | compression profile | Improved |
Closest papers by meaning
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.87)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.87)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.86)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.86)
- MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search (NeurIPS 2026 · similarity 0.86)
- Maestro: Uncovering Low-Rank Structures via Trainable Decomposition (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
