Research map / Architectures and efficiency / Pruning and efficient tuning
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRAM, which trains LoRA adapters on a pruned LLM, recovers them to the original shape, and applies them to the full model for inference, with optional continual pre-training alignment and 4-bit quantization (QLoRAM).
Problem
LoRA fine-tuning of large LLMs remains memory-bound by storing the frozen base model weights, even when quantized to 4 bits.
Approach
Train low-rank adapters on a pruned (smaller) base model, then recover them by zero-filling pruned positions and merge into the original unpruned model for inference; a one-time continual pre-training of the pruned model aligns its knowledge with the original, and QLoRA-style quantization is applied to the pruned base.
Key results
- QLoRAM-Stru on LLaMA-2-70B reaches 12.84x parameter reduction and raises GSM8K from 52.01% (70B w/o FT) to about 57%, versus 37% for LoRA-trained 13B.
- QLoRAM-Stru on LLaMA-3.1-70B achieves a 15.81x parameter reduction with perplexity between LoRA-trained 8B and LoRA-trained 70B.
- At a 28.56x reduction, QLoRAM-Stru keeps perplexity around 2.5 while naive pruning rises to 621.98.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train LoRA updates on a pruned model and recover the updates for inference | LoRAM · GSM8K, OpenHermes | accuracy: 52.01 to 57.16 | Helped |
| Add alignment by continual pre-training the pruned model (ablation) | LoRAM · Alpaca test set, LLaMA-2-13B | perplexity | Helped |
| Add recovery of the trained pruned low-rank matrices (ablation) | LoRAM · Alpaca test set, LLaMA-2-13B | perplexity | Helped |
| Use non-structured rather than structured pruning | LoRAM · In-domain OpenHermes and OpenOrca tests | perplexity | Helped |
| Use non-structured rather than structured pruning | LoRAM · Out-of-domain Alpaca tests | perplexity | No clear effect |
| Use selective structured pruning instead of random pruning | LoRAM · LLaMA-2-70B downstream tasks | task performance | Helped |
| Increase parameter reduction from 9.82× toward 16.95× | QLoRAM-Stru · OpenHermes downstream tasks | task performance | Helped |
| Increase parameter reduction to 28.56× | QLoRAM-Stru · Downstream tasks, especially code generation | task performance | Hurt |
Surprising findings
- Training LoRA on a pruned model and merging the recovered adapters into the unpruned model can outperform LoRA trained directly on a larger unpruned model in downstream tasks.
- Non-structured variants (LoRAM-Semi/Unst) achieve better in-domain perplexity but their theoretical reduction is not realized in actual memory, since pruned weights are zero-filled.
Limitations
- Non-structured pruning yields no actual memory reduction because pruned weights are stored as zeros.
- Downstream gains depend on task and pruning ratio; overly aggressive pruning (28.56x) leaves too few neurons for code generation.
- Results on OpenOrca are sensitive to learning rate, requiring per-baseline tuning.
Experimental setup
Datasets: FineWeb OpenWebMath OpenHermes-2.5 OpenOrca Alpaca MathQA GSM8K ARC HellaSwag OpenBookQA PIQA WinoGrande
Benchmarks: MathQA GSM8K ARC Challenge ARC Easy HellaSwag OpenBookQA PIQA WinoGrande HumanEval
Metrics: accuracy perplexity Pass@1 Pass@10 parameter reduction ratio
Baselines: LoRA on LLaMA-2-7B LoRA on LLaMA-2-13B LoRA on LLaMA-3.1-8B LoRAM-Rand LoRAM-Semi LoRAM-Unst naive pruning
Builds on: LoRA QLoRA LLM-Pruner SparseGPT
Closest papers by meaning
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.92)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.91)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.