Research map / Large language models / Instruction and fine-tuning
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune quantization scales while keeping integer weights frozen instead of applying LoRA plus OPTQ | LLaMA-7B · Wikitext2, 3-bit | perplexity: 19.47 -> 6.19 | Helped |
| Fine-tune quantization scales while keeping integer weights frozen instead of applying LoRA plus OPTQ | LLaMA-7B · PTB, 4-bit | perplexity: 11.22 -> 9.69 | Helped |
| Instruction-tune the quantized model by updating PEQA scales after RTN | LLaMA-7B · MMLU five-shot | accuracy: 28.3 -> 35.8 | Helped |
| Instruction-tune the quantized model by updating PEQA scales after RTN | LLaMA2-70B · MMLU five-shot | accuracy: 67.0 -> 67.5 | Helped |
| Use PEQA instead of LoRA w/ OPTQ | LLaMA-7B · Natural Instruction zero-shot | ROUGE-L: 25.0 -> 27.1 | Helped |
| Learn zero-points only instead of learning quantization scales (ablation) | PEQA on LLaMA-7B · Wikitext2, 4-bit | perplexity: 5.84 -> 11.56 | Hurt |
| Learn both zero-points and quantization scales instead of scales only (ablation) | PEQA on LLaMA-7B · Wikitext2, 4-bit | perplexity: 5.84 -> 5.86 | No clear effect |
| Add grouped learnable scales using g64 rather than channel-wise scales (ablation) | PEQA on LLaMA-7B · Wikitext2, 4-bit | perplexity: 5.84 -> 5.64 | Helped |
| Add grouped learnable scales using g64 rather than channel-wise scales (ablation) | PEQA on LLaMA-13B · Wikitext2, 3-bit | perplexity: 5.54 -> 5.34 | Helped |
| Use LoRA QKVO16 rather than LoRA QV4 (ablation) | LoRA on LLaMA-7B · Wikitext2 | perplexity: 5.53 -> 5.50 | No clear effect |
| Increase quantization granularity by adding more learnable group scales (ablation) | PEQA · Wikitext2 | perplexity | Helped |
| Use PEQA instead of full-precision LoRA | LLaMA-7B · Wikitext2, 4-bit | perplexity: 5.53 -> 5.84 | Hurt |
Closest papers by meaning
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.91)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.91)
- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.91)
- LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.