Research map / Architectures and efficiency / Quantization
LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoTA-QAF, a quantization-aware fine-tuning method that uses ternary adapters and a sign-based optimizer (t-SignSGD) to adjust all quantized weights in-grid and merge losslessly into GPTQ-quantized LLMs.
Problem
Fine-tuning quantized LLMs with high-precision adapters (e.g., LoRA) causes data-type mismatch at inference and accuracy loss when adapters are merged into low-bit weights, while existing lossless merges (e.g., QA-LoRA) only adjust zero factors rather than the quantized weights.
Approach
Ternary adapters A_T, B_T in {-1,0,1} form an auxiliary matrix whose thresholded ternary version is added directly to the integer quantized weights, with an offset absorbed into the zero factor, enabling lossless merging; t-SignSGD updates the ternary adapters using sign updates gated by a dynamic top-percentile gradient threshold.
Key results
- On MMLU, LoTA-QAF surpasses 16-bit LoRA by up to 5.14% (Qwen 2.5 14B, 2-bit), reaching 47.80 average vs 42.66.
- Task-specific fine-tuning: LoTA-QAF outperforms QA-LoRA but trails GPTQ+LoRA across GSM8K, SQL, and ViGGO.
- After merging, LoTA inference is 1.7x-2.0x faster than LoRA with 16-bit adapters (Llama 3.1 8B, ViGGO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoTA-QAF instead of GPTQ+LoRA for 2-bit quantization | Qwen 2.5 14B · MMLU (5-shot) | accuracy: 42.66 -> 47.80 | Helped |
| Use LoTA-QAF instead of GPTQ+LoRA for 2-bit quantization | Qwen 2.5 32B · MMLU (5-shot) | accuracy: 48.46 -> 52.65 | Helped |
| Use LoTA-QAF instead of GPTQ+LoRA | Quantized models across tested sizes and bit widths · MMLU (5-shot), Avg. | accuracy | Helped |
| Use LoTA-QAF instead of QA-LoRA | Quantized models · Task-specific GSM8K, SQL, and ViGGO | accuracy | Helped |
| Use LoTA-QAF instead of 16-bit-adapter LoRA after merging | Llama 3.1 8B · ViGGO inference, 4-, 3-, and 2-bit | throughput speedup: 1.9x, 1.7x, and 2.0x | Helped |
| Use LoTA instead of LoRA | Llama 3.1 8B · SQL generation, 2-bit | convergence loss: 0.132 -> 0.375 | Hurt |
| Use LoTA instead of LoRA | Llama 3.1 8B · SQL generation, 4- and 3-bit | convergence loss difference: <0.01 | No clear effect |
| Implement ternary adapters with LoTA-QAF rather than LoRA | Llama 3.1 8B, 4-bit · Training across four datasets | training time: 14.1%-25.4% more training time | Hurt |
| Implement ternary adapters with LoTA-QAF rather than LoRA | Llama 3.1 8B, 4-bit · Training across four datasets | peak memory usage: 2.6%-6.3% overhead | Hurt |
| Set TA threshold ω to 48 rather than the other tested values (ablation) | Llama 3.1 8B · GSM8K, 4-, 3-, and 2-bit | accuracy | Helped |
| Use a larger ω to make quantized-weight adjustments more conservative (ablation) | Llama 3.1 8B · GSM8K, 2-bit | accuracy | Helped |
| Use a smaller ω (ablation) | Llama 3.1 8B · GSM8K, 3- and 2-bit | accuracy | Hurt |
Surprising findings
- LoTA-QAF's 2-bit 14B MMLU average (47.80) exceeds the 16-bit LoRA variant, despite the ternary adapter's restricted values.
- Training is 14.1%-25.4% slower than LoRA because ternary adapters are simulated in bfloat16 in PyTorch.
Limitations
- Training is slower than LoRA due to simulated ternary dtype and boundary checks; no native ternary kernel.
- Convergence is worse than LoRA at 2-bit (SQL loss 0.375 vs 0.132), and task-specific results trail 16-bit LoRA.
- Hyperparameters (threshold omega, percentile sigma_t) require tuning per model, bit-width, and dataset.
Experimental setup
Datasets: Alpaca GSM8K SQL generation ViGGO C4
Benchmarks: MMLU GSM8K SQL generation ViGGO
Metrics: accuracy throughput (tokens/s) peak memory training time
Baselines: GPTQ+LoRA QA-LoRA GPTQ
Builds on: LoRA QA-LoRA QLoRA GPTQ AutoRound
Closest papers by meaning
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.91)
- QERA: an Analytical Framework for Quantization Error Reconstruction (ICLR 2025 · similarity 0.89)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.88)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.88)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.88)
- OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.