Research map / Architectures and efficiency / Quantization

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

ICLR 2025 · Architectures and efficiency › Quantization · approach: KV cache quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Dobi-SVD compresses LLMs by directly truncating activations with a differentiable, gradient-stabilized truncation-position search, IPCA-based optimal weight update, and quantized remapping that makes compression ratio bijective with truncation position.

Problem

SVD-based LLM compression degrades sharply at high compression ratios because weight-truncation or activation-aware SVD methods pick suboptimal truncation positions, update weights poorly, and suffer from the limitation that real compression requires discarding at least half the singular values.

Approach

Truncate activations (A = xW) via SVD instead of weights, learn per-matrix continuous truncation positions by differentiable tanh-smoothed truncation with Taylor-stabilized SVD backprop, derive the rank-k weight update by IPCA on projected weights, and remap storage by 8-bit quantizing the orthogonal U and V factors so compression ratio maps bijectively to k.

Key results

• At 0.4 compression ratio on LLaMA-7B, Dobi-SVD reaches 9.95 WikiText2 perplexity (remapped), versus 53.74 for SVD-LLM (LoRA) and 57057 for ASVD.
• On a 12GB TITAN Xp, the compressed LLaMA-7B at 0.4 ratio reaches 25.97 tokens/s, a 12.4x speedup over the original (2.09 tokens/s), since the model fits fully on GPU.
• On LLaVA-v1.5 at 0.4 ratio, Pope-random accuracy rises to 89.8 from 88.1 baseline, and throughput improves by up to 11.9% at batch size 1.

Measured effects reported in the paper

Surprising findings

• Truncating only the later attention layers (e.g., layer 29 alone) with differentiable training yields performance better than the original uncompressed model on the reported metric.
• Without remapping, Dobi-SVD at 0.4 ratio gives 58.02 WikiText2 perplexity, showing the storage bijection is essential at low ratios.

Limitations

• SVD implementation is time- and memory-intensive because only fp32 SVD is supported in Python.
• Transitioning precision in remapping becomes hard below 4-bit, with a performance drop at 2-bit.
• Quantization adds dequantization overhead during inference.

Experimental setup

Datasets: WikiText2 C4 PTB TextQA BridgeData V2 OpenbookQA WinoGrande HellaSwag PIQA MathQA ARC-e ARC-c

Benchmarks: WikiText2 PTB C4 PIQA HellaSwag WinoGrande ARC-e ARC-c OpenbookQA MathQA MMLU BoolQ

Metrics: perplexity accuracy tokens/s memory (GB) MSE

Baselines: ASVD SVD-LLM LLM-Pruner FLAP Wanda-sp SliceGPT Bonsai GPTQ

Builds on: SVD-LLM ASVD Eckart-Young-Mirsky theorem Incremental PCA GPTQ

Closest papers by meaning

• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.93)
• LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.91)
• Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (ICLR 2025 · similarity 0.91)
• MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition (ICML 2025 · similarity 0.89)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.89)
• 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.88)