Research map / Architectures and efficiency / Quantization
ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression
Wenya Yu, Chao Zhang, Li Wang, Samson Lasaulce, Merouane Abdelkader DEBBAH
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ProjQ instead of LoftQ | LLaMA2-7B 2-bit · C4 | perplexity: 28.77 -> 21.50 | Improved |
| use ProjQ instead of GPTQ+SVD-LLM | LLaMA2-7B 2-bit · Common Sense avg | accuracy (%): 45.33 -> 47.82 | Improved |
| use ProjQ instead of LoftQ before fine-tuning | LLaMA2-7B 2-bit · WikiText-2 fine-tuned | perplexity: 9.14 -> 8.17 | Improved |
| use ProjQ instead of LoftQ | LLaMA2-7B 2-bit · Common Sense fine-tuned | accuracy (%): 54.78 -> 57.59 | Improved |
| use ProjQ instead of LoftQ | LLaMA2-7B 3-bit · GSM8K fine-tuned | accuracy (%): 37.68 -> 35.18 | Hurt |
| use ProjQ instead of LoftQ | Qwen2.5-14B-Ins 2-bit · GSM8K fine-tuned | accuracy (%): 51.25 -> 51.11 | No clear effect |
| use ProjQ instead of QLoRA | Qwen2.5-7B-Ins 4-bit · Common Sense fine-tuned | accuracy (%): 71.59 -> 66.78 | Hurt |
| add proposed adapter initialization over standard LoRA init (ablation) | ProjQ LLaMA2-7B · WikiText-2 fine-tuned | perplexity: 9.17 -> 8.17 | Improved |
| add proposed adapter initialization over standard LoRA init (ablation) | ProjQ Qwen2.5-7B-Ins · Common Sense avg | accuracy (%): 53.66 -> 59.88 | Improved |
| increase alternating iterations from 1 to 3 (ablation) | ProjQ LLaMA2-7B · C4 | perplexity: 25.62 -> 25.32 | No clear effect |
| increase alternating iterations from 1 to 2 (ablation) | ProjQ LLaMA2-7B · Common Sense avg | accuracy (%): 46.91 -> 40.91 | Hurt |
| set designed rank r_d equal to adapter rank 64 instead of 16 (ablation) | ProjQ LLaMA2-7B · WikiText (zero-shot) | perplexity: 29.07 -> 22.42 | Improved |
Closest papers by meaning
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- LQER: Low-Rank Quantization Error Reconstruction for LLMs (ICML 2024 · similarity 0.91)
- ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.91)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.91)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.91)
- Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
