AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Mitigating Error Propagation in Low-Rank Approximation of Large Models via Distribution-Aware Whitening

Yao Zhu, Rui Sun, Xiu Yan, Yang Li, Wang Lu, Yunjian Zhang

ICML 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
re-estimate whitening from compressed-layer inputs (distribution-aware) instead of static whiteningSVD-LLM style compression · WikiText-2, ratio 0.3perplexity: 33.176 -> 23.702 on LLaMA3-8BImproved
distribution-aware whitening on LLaMA-13B at ratio 0.3 on WikiText-2SVD-LLM style compression · WikiText-2, ratio 0.3, LLaMA-13Bperplexity: 7.618 -> 7.635No clear effect
distribution-aware whitening at higher compression ratioSVD-LLM style compression · WikiText-2, ratio 0.5, LLaMA3-8Bperplexity: 3908.397 -> 83.950Improved
apply distribution-aware compression before GPTQ quantizationcompressed Vicuna-v1.5-7B · C4, GPTQ-4bitperplexity: SVD-LLM 85.03, 34.77 worse than oursImproved
apply distribution-aware compression before GPTQ 8-bitcompressed Vicuna-v1.5-7B · C4, GPTQ-8bitperplexity: SVD-LLM only 0.31 higherNo clear effect
add post-compression fine-tuning (full update) on top of distribution-aware compressioncompressed LLaMA3-8B · PTB, ratio 0.5perplexity: 128.2454 -> 64.5531 vs SVD-LLMImproved
use distribution-aware vs PiSSA initialization for LoRALoRA fine-tuning, Vicuna-V1.5-7B · GSM8K, rank 256accuracy: 0.320 -> 0.384Improved
use distribution-aware vs PiSSA initialization for LoRALoRA fine-tuning, Qwen2-7B · Object Counting, rank 256accuracy: 0.680 -> 0.848Improved
use distribution-aware vs PiSSA initialization for LoRA at rank 64LoRA fine-tuning, Vicuna-V1.5-7B · HumanEval and HumanEval+pass rate: 0.183 -> 0.183; 0.165 -> 0.165No clear effect
use distribution-aware init instead of Kaiming init for LoRALoRA fine-tuning, Vicuna-V1.5-7B · MetaMathQA / Code Feedback training losstraining lossImproved
compare against pruning methods (FLAP, LLM-Pruner) after fine-tuningVicuna-v1.5-7B compressed · C4perplexityMixed

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy