Research map / Large language models / Instruction and fine-tuning
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AmoebaLLM, a one-time fine-tuning framework that lets a pretrained LLM instantly yield depth- and width-shrunk subnets via DP-based layer selection, importance-based width pruning, a shape-aware mixture of LoRAs, and loss-magnitude-balanced in-place distillation.
Problem
Deploying pretrained LLMs efficiently across platforms with different resource constraints requires subnets of varied depth and width, but one-for-all training methods fail on pretrained LLMs and per-platform fine-tuning is costly.
Approach
Select knowledge-preserving layers via dynamic programming and neurons via importance scores, then one-for-all fine-tune with a shape-conditioned gated mixture of LoRAs (mergeable at deployment) and in-place distillation whose subnet losses are magnitude-normalized to the largest subnet.
Key results
- On LLaMA2 7B at 50% remaining ratio, AmoebaLLM reaches 30.60 MMLU and 52.19 average commonsense accuracy, versus 27.67 MMLU and 51.12 average for FLAP.
- DP-based depth shrinking at 18 remaining layers gives 43.1% MMLU and 38.70 Wikitext2 PPL without fine-tuning, a +9.4% MMLU gain over the strongest baseline.
- The SMoL adapter raises the largest subnet's MMLU by 6.6% over standard LoRA in the depth-only setting (46.6 vs 40.6 for the 32-layer model, LoRA 40.6 vs SMoL 47.2 in full model table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AmoebaLLM instead of FLAP at 80% remaining | LLaMA2 7B compression · MMLU, 80% remaining | accuracy: 40.21 -> 40.70 | Helped |
| use AmoebaLLM instead of FLAP at 50% remaining | LLaMA2 7B compression · MMLU, 50% remaining | accuracy: 27.67 -> 30.60 | Helped |
| use AmoebaLLM instead of the strongest baseline at 80% remaining | Vicuna 7B v1.5 compression · MMLU, 80% remaining | accuracy: 43.50 -> 47.40 | Helped |
| use AmoebaLLM instead of FLAP at 65% remaining | Vicuna 7B v1.5 compression · Average commonsense, 65% remaining | accuracy: 56.47 -> 55.93 | Hurt |
| use AmoebaLLM instead of per-subnet fine-tuning | LLaMA2 7B, 32 layers · MMLU | accuracy: 46.4 -> 47.2 | Helped |
| replace full-model fine-tuning with SMoL (ablation) | AmoebaLLM on LLaMA2 7B, 24 layers · MMLU | accuracy: 32.6 -> 45.1 | Helped |
| replace standard LoRA with SMoL (ablation) | AmoebaLLM on LLaMA2 7B, 32 layers · MMLU | accuracy: 40.6 -> 47.2 | Helped |
| add loss-magnitude balancing (ablation) | AmoebaLLM on LLaMA2 7B, 32 layers · MMLU | accuracy: 42.0 -> 47.2 | Helped |
| use DP-based layer selection instead of Shortened LLaMA (ablation) | LLaMA2 7B layer pruning, 18 layers retained · Wikitext2 | perplexity: 71.80 -> 38.70 | Helped |
| use DP-based layer selection instead of Shortened LLaMA (ablation) | LLaMA2 7B layer pruning, 18 layers retained · MMLU | accuracy: 33.7 -> 43.1 | Helped |
| calibrate layer selection on MMLU rather than Wikitext2 (ablation) | LLaMA2 7B depth shrinking, 20 layers · MMLU | accuracy: 24.4 -> 41.2 | Helped |
| calibrate layer selection on MMLU rather than Wikitext2 (ablation) | LLaMA2 7B depth shrinking, 20 layers, after fine-tuning · MMLU | accuracy: 23.6 -> 41.0 | Helped |
Surprising findings
- Full-model fine-tuning on the small Alpaca corpus gives large PPL degradation (e.g., 167.74 at 20 layers without SMoL, a full-model variant), showing instability.
- MMLU-calibrated layer selection gives worse Wikitext2 PPL before fine-tuning but recovers language modeling after fine-tuning, suggesting factual knowledge is harder to restore than language modeling ability.
Limitations
- Only parameter-efficient fine-tuning on a small dataset (Alpaca) was used, limiting the achievable accuracy-efficiency trade-off.
- Width importance scores and DP calibration rely on a small calibration set (MMLU training data), and the approach is evaluated only on two 7B models.
- Subnet search is a simple grid search; more advanced search such as evolutionary search is left to future work.
Experimental setup
Datasets: Alpaca MMLU Wikitext2 BookCorpus
Benchmarks: MMLU BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA
Metrics: accuracy average zero-shot accuracy latency perplexity
Baselines: LLM-Pruner FLAP Shortened LLaMA Unreasonable layer pruning
Builds on: BigNAS sandwich sampling in-place distillation noisy top-K gating LLM-Pruner FLAP
Closest papers by meaning
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.89)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.89)
- $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.89)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.89)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.