Research map / Architectures and efficiency / Quantization
FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FALQON, which melds LoRA adapters into an FP8-quantized LLaMA backbone during fine-tuning (quantization error approximated by a low-rank SVD, A frozen and concatenated to the weights, B updated through a row-wise proxy buffer), giving about 3x faster LoRA fine-tuning than quantized LoRA baselines at similar MMLU accuracy.
Problem
FP8 quantization overhead outweighs its speedup for the small-dimensional matrix multiplications in LoRA, so quantized LoRA fine-tuning is slow despite FP8 hardware support.
Approach
Treat the FP8 quantization error of the backbone as an implicit low-rank LoRA adapter (via SVD), merge it into the quantized backbone so the adapter needs no separate path, compute the B gradient from an intermediate Ax obtained in the same quantized matmul, and apply only top-k row updates from a proxy buffer to the FP8 weights.
Key results
- On LLaMA-7B (Alpaca), FALQON runs at 1.80 s/step versus 5.45 s for QLoRA and 9.44 s for QA-LoRA (about 3x), with average MMLU 0.3491 versus 0.3272 for QLoRA.
- On LLaMA-13B, FALQON runs at 3.26 s/step versus 18.02 s for QA-LoRA on Alpaca, while TorchAO and Fishman et al. run out of memory on a single RTX 4090.
- On commonsense reasoning with LLaMA-7B (Alpaca), FALQON averages 0.6320, below LoRA FP16 (0.6464), QLoRA (0.6438) and TorchAO (0.6455).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| merge LoRA adapters into the FP8 backbone during fine-tuning | Quantized LoRA fine-tuning · Alpaca, LLaMA-7B | time per step: 9.44 vs. 1.80s | Helped |
| merge LoRA adapters into the FP8 backbone during fine-tuning | Quantized LoRA fine-tuning · Alpaca, LLaMA-13B | time per step: 18.02 vs. 3.26s | Helped |
| use FALQON instead of QA-LoRA | Quantized LoRA fine-tuning · Alpaca, LLaMA-7B | MMLU average: 0.3548 -> 0.3491 | Hurt |
| use FALQON instead of QA-LoRA | Quantized LoRA fine-tuning · Alpaca, LLaMA-13B | MMLU average: 0.4729 -> 0.4644 | Hurt |
| use FALQON instead of QA-LoRA | Quantized LoRA fine-tuning · OASST1, LLaMA-7B | MMLU average: 0.3609 -> 0.3481 | Hurt |
| use FALQON instead of QA-LoRA | Quantized LoRA fine-tuning · OASST1, LLaMA-13B | MMLU average: 0.4769 -> 0.4645 | Hurt |
| update matrix A (ablation) | FALQON · Alpaca, LLaMA-7B | time per step: 1.80 -> 1.94 | Hurt |
| update matrix A (ablation) | FALQON · Alpaca, LLaMA-7B | MMLU: 0.3491 -> 0.3469 | No clear effect |
| update matrix A (ablation) | FALQON · OASST1, LLaMA-7B | MMLU: 0.3481 -> 0.3470 | No clear effect |
| update matrix A (ablation) | FALQON · Alpaca, LLaMA-13B | time per step: 3.26 -> 3.89 | Hurt |
| update matrix A (ablation) | FALQON · Alpaca, LLaMA-13B | MMLU: 0.4644 -> 0.4634 | No clear effect |
| update matrix A (ablation) | FALQON · OASST1, LLaMA-13B | MMLU: 0.4645 -> 0.4656 | No clear effect |
Surprising findings
- Standard FP8 LoRA runs about half the throughput of FP16 LoRA across typical ranks (16-128), so quantization overhead cancels the expected FP8 speedup.
- FALQON's commonsense reasoning average is lower than several baselines, including QLoRA and TorchAO FP8, on LLaMA-7B Alpaca.
Limitations
- Validated only on decoder-only LLaMA Transformers (7B and 13B); not tested on encoder models, diffusion models, or 70B-scale models.
- Commonsense reasoning accuracy is slightly below several quantized LoRA baselines on some tasks (e.g., OBQA, ARC-C).
- Freezing matrix A and approximating the quantization error with SVD are heuristic choices; the paper's analysis of top-k sensitivity is limited to MMLU.
Experimental setup
Datasets: Alpaca OASST1
Benchmarks: MMLU HellaSwag PIQA WinoGrande ARC BoolQ OpenBookQA
Metrics: accuracy time per step (s) trainable parameters training cost (USD)
Baselines: QLoRA QA-LoRA IR-QLoRA FP6-LLM TorchAO Fishman et al. FP8 training LoRA FP16
Builds on: LoRA QLoRA LoRA-FA TorchAO
Closest papers by meaning
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.90)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.89)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.89)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.88)
- HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.