Research map / Architectures and efficiency / Pruning and efficient tuning

CLOVER: Cross-Layer Orthogonal Vectors Pruning

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CLOVER, which applies SVD to the merged Q-K and V-O pairs within each attention head to orthogonalize them, enabling higher-ratio structured pruning and parameter-efficient fine-tuning of singular values.

Problem

Decoder-only LLM inference is memory-bound because of the growing key/value cache, and structured pruning of attention head dimensions degrades performance because head vectors carry linear redundancy.

Approach

Treat the Q-K and V-O pairs in each attention head as low-rank factorizations of W_QK and W_VO, and apply SVD across the two layers so the head's vectors become orthogonal; singular values rank dimensions for pruning and serve as the only trainable parameters for fine-tuning, then merge back into the weights without extra parameters.

Key results

• On LLaMA-2-7B, CLOVER PEFT reaches 85.2 average accuracy on eight commonsense tasks, above PiSSA (84.5), HiRA (81.4), DoRA (79.7), and LoRA (77.6).
• On GPT-2-XL, pruning 70% of the attention Q-K dimension with CLOVER gives perplexity comparable to vanilla pruning at 8%, per the abstract; with 50% pruning, CLOVER fine-tuned reaches 17.22 perplexity vs 20.45 without fine-tuning on the full-training variant at 66M tokens (vanilla 35.12 before fine-
• CLOVER with TransMLA reaches up to 11.1x speedup over LLaMA-2-7B at 8K context on a 24GB GPU in vLLM simulations.

Measured effects reported in the paper

Surprising findings

• Whisper-large-v3 retains nearly unchanged output after training-free CLOVER pruning of 46.42% of attention parameters, while vanilla pruning at the same ratio fails to produce valid output.
• Pruning 50% of the head dimension in DeepSeek-V2-Lite with CLOVER (59.84 vs 61.54 full average at -25%) and SliceGPT differ sharply in latency, with CLOVER giving a speedup while SliceGPT does not.

Limitations

• Cross-layer orthogonalization is not feasible when nonlinear operations such as RoPE sit between Q-K or V-O pairs, requiring head-wise orthogonalization in the Key layer instead.
• Evaluation focuses on basic pruning and does not compare against other state-of-the-art pruning methods.

Experimental setup

Datasets: Commonsense-148k OpenWebText WikiText-2 LibriSpeech Long mixed pretraining dataset

Benchmarks: MMLU ARC-e ARC-c PIQA HellaSwag Winogrande BoolQ SIQA OBQA WikiText-2

Metrics: accuracy perplexity speedup latency peak GPU memory

Baselines: SliceGPT TransMLA LoRA DoRA HiRA PiSSA SVFT Vanilla pruning

Builds on: PiSSA SVFT TransMLA SliceGPT LoRA

Closest papers by meaning

• When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.88)
• When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA (NeurIPS 2026 · similarity 0.87)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.87)
• An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.87)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.87)
• Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.87)