Research map / Large language models / Instruction and fine-tuning
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LowRA, a data-free post-training quantization framework for LoRA fine-tuning of LLMs below 2 bits per parameter, combining weighted Lloyd-Max mapping/threshold search, a two-level ILP channelwise precision assigner, and CUDA quantize/dequantize kernels.
Problem
Quantized LoRA fine-tuning has been limited to 2-4 bits per parameter, and existing methods use coarse-grained precision, globally shared data formats, and simulated (fake) quantization, which blocks ultra-low-bit LoRA fine-tuning on resource-constrained hardware.
Approach
Use per-output-channel mixed precision (1/2/4-bit) chosen by a hierarchical ILP that minimizes summed square error under a global bit budget, pair it with per-channel weighted Lloyd-Max mapping/threshold search over absmax-normalized groups, initialize LoRA adapters with LoftQ, and implement real (not simulated) CUDA quantization kernels.
Key results
- At 2-bit on WikiText-2 (LLaMA-2-7B), LowRA reaches perplexity 6.60 vs 9.17 for QLoRA and 8.63 for LoftQ.
- LowRA fine-tunes LLaMA-2-7B at 1.75 bits (perplexity 7.76 on WikiText-2) and LLaMA-30B at 1.15 bits (perplexity 8.00 on WikiText-2).
- Reports up to 50% memory reduction for LLaMA-30B inference and enables LLaMA-30B fine-tuning on a single 16GB T4 GPU (claimed).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LowRA instead of LoftQ at 2 bits | LLaMA-2-7B · WikiText-2 | perplexity: 8.63 -> 6.60 | Helped |
| use LowRA instead of LoftQ at 2 bits | LLaMA-2-7B · OASST1 | perplexity: 5.68 -> 4.35 | Helped |
| use LowRA instead of LoftQ at 2 bits | LLaMA-2-13B · WikiText-2 | perplexity: 7.27 -> 5.79 | Helped |
| use LowRA instead of LoftQ at 2 bits | LLaMA-2-13B · OASST1 | perplexity: 4.75 -> 3.84 | Helped |
| use LowRA instead of LoftQ at 2 bits | BART-large · XSUM | ROUGE-1: 31.89 -> 36.75 | Helped |
| use LowRA instead of LoftQ at 2 bits | BART-large · CNN/DailyMail | ROUGE-1: 38.88 -> 40.15 | Helped |
| increase precision from 2 bits to 4 bits with LowRA | LLaMA-2-7B · WikiText-2 | perplexity: 6.60 -> 5.25 | Helped |
| add mapping and threshold search to the precision assigner (ablation) | LLaMA-2-7B · WikiText-2, 2-bit | perplexity: 8.63 -> 6.60 | Helped |
| add mapping and threshold search to the precision assigner (ablation) | LLaMA-2-7B · WikiText-2, 2-bit | accuracy: 0.536 -> 0.574 | Helped |
| add the precision assigner to LoftQ (ablation) | BART-large · XSUM, 2-bit | ROUGE-1: 31.8941 -> 31.8941 | No clear effect |
| add the precision assigner to LoftQ (ablation) | LLaMA-2-7B · WikiText-2, 2-bit | perplexity: 8.63 -> 8.63 | No clear effect |
| replace LoftQ initialization with PiSSA (ablation) | LLaMA-7B · WikiText-2, 2-bit | perplexity: 8.63 -> 1919.63 | Hurt |
Surprising findings
- PiSSA initialization fails at low bit widths (perplexity ~1919 at 2-bit on LLaMA-7B), while LoftQ works, reversing the expectation that PiSSA converges faster.
- QLoRA produces DNC (non-convergence) on XSUM at 2 bits and near-zero ROUGE-2 on CNN/DailyMail at 2 bits.
Limitations
- Experiments limited to LLaMA-2 family, LLaMA-30B and BART-large; no evaluation on newer architectures.
- Reported results at 1.15-1.75 bits rely on only a few tasks and single-run comparisons.
- Shared thresholds and mappings across adapters; task-specific mapping learning is left as costlier extension.
Experimental setup
Datasets: WikiText-2 OASST1 XSUM CNN/DailyMail
Benchmarks: WikiText-2 OASST1 XSUM CNN/DailyMail
Metrics: perplexity accuracy ROUGE-1 ROUGE-2 ROUGE-L memory footprint
Baselines: QLoRA LoftQ PiSSA
Builds on: QLoRA LoftQ LoRA PiSSA NormalFloat
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.91)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.90)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.90)
- LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.