Research map / Architectures and efficiency / Pruning and efficient tuning

La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LaRoSA, a training-free activation sparsification method that combines layerwise PCA-based orthogonal rotations with Top-K selection to give consistent model-level sparsity and wall-clock speed-ups on non-ReLU LLMs.

Problem

Activation sparsity for non-ReLU LLMs (e.g., SwiGLU-based) either requires recovery training or relies on magnitude-based pruning with offline thresholds, which yields fluctuating model-level sparsity and unstable inference speed-ups.

Approach

Apply a layer-specific orthogonal rotation (PCA of input activation covariance, with residual adapters so rotations are absorbed into weights) before a per-input Top-K selection, replacing magnitude thresholds so each input gets a fixed sparsity level, plus grid-searched per-activation-type allocation coefficients.

Key results

• On LLaMA2-7B at 40% sparsity, LaRoSA reaches 5.64 PPL vs dense 5.47 and TEAL 6.40, with 66.15 zero-shot average vs TEAL 64.92 and CATS 49.55.
• LaRoSA achieves 1.30x single-batch speed-up on LLaMA3-8B at 50% sparsity on A100 and up to 1.45x on LLaMA3-70B at 50% (H20 1.41x for 8B).
• On DS-R1-Distill-Qwen2.5-7B at 25% sparsity, LaRoSA keeps MATH-500 at 91.0 vs 93.4 reproduced dense.

Measured effects reported in the paper

Surprising findings

• Offline magnitude thresholds calibrated on tokens often mismatch the actual thresholds needed, especially for initial tokens of a sequence, explaining TEAL's need to skip 1% of prefill tokens.
• Calibration dataset choice (WikiText2, Alpaca, C4) has minimal effect on LaRoSA performance because covariance matrices across datasets have cosine similarity near 1.

Limitations

• Speed-up at 0% sparsity is below dense because of residual adapter and sparsification overhead.
• Full sparsification at prefill is applied to all methods for fairness, departing from TEAL's reported setting.
• Some per-task results in the 70B appendix table appear inconsistent (e.g., LLaMA2-70B 25% WinoGrande 57.17 and BoolQ 48.60), suggesting possible reporting errors.

Experimental setup

Datasets: WikiText2 Alpaca C4

Benchmarks: WikiText2 perplexity ARC-Easy ARC-Challenge PIQA BoolQ HellaSwag OpenBookQA WinoGrande MMLU MATH-500 GPQA-Diamond AIME 2024

Metrics: perplexity accuracy tokens/sec speed-up actual sparsity

Baselines: CATS TEAL TopK TopK+GS

Builds on: TEAL CATS SliceGPT QuaRot DejaVu

Closest papers by meaning

• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.94)
• Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.92)
• Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.92)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.92)
• SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.92)
• Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.92)