Research map / Architectures and efficiency / Pruning and efficient tuning

ESPACE: Dimensionality Reduction of Activations for Model Compression

NeurIPS 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Joint pruning and quantization · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ESPACE, which compresses LLMs by statically projecting activations onto pre-calibrated principal components (eigenvectors of activation autocorrelation), then retrains the uncompressed weights to heal the approximation error.

Problem

Existing LLM tensor decomposition methods compress frozen weights, which breaks weight structure, reduces trainable parameters and hurts expressivity and retraining; activation decomposition has been unexplored because activations are dynamic at inference.

Approach

Insert a static orthonormal projection P (top eigenvectors of calibrated activation autocorrelation, or variants minimizing forward-propagated error bounds) before each GEMM; by matrix multiplication associativity, the weight becomes P^T W, so inference compression comes from precomputing it while all weights remain trainable and fully available for retraining.

Key results

• ESPACE compresses GPT3-22B by 40% to WK-103 perplexity 6.29 vs 6.55 baseline, and at 55% to 6.73 with small accuracy loss.
• ESPACE at 20% compression lowers GPT3-8B perplexity from 7.38 to 7.00 (0.38 decrease) while also improving downstream accuracy average to 62.9 vs 62.1.
• GEMM latency reduced by up to 49% and time to first token by up to 43% on A100 for GPT3 models.

Measured effects reported in the paper

Surprising findings

• At 20-40% compression, ESPACE-compressed GPT3 models outperform their uncompressed baselines in perplexity and accuracy, suggesting a regularization effect.
• Out-of-the-box GPT3-22B at 20% compression is nearly iso-accurate without retraining (perplexity 6.61 vs 6.55).

Limitations

• Retraining requires hundreds of billions of tokens, which is expensive.
• Only GEMM layers are compressed; attention cross-activation products are untouched and end-to-end serving throughput was not evaluated.
• Comparison with related works is limited to Llama2-7B and relies on perplexity increase rather than matched settings.

Experimental setup

Datasets: Wikitext-103 MTNLG dataset

Benchmarks: Wikitext-103 BoolQ HellaSwag PIQA RACE WinoGrande

Metrics: perplexity zero-shot accuracy GEMM latency time to first token

Baselines: ASVD SliceGPT SVD-LoRa Llama2-7B baseline GPT3 baseline

Builds on: PCA SliceGPT ASVD Megatron-LM

Closest papers by meaning

• 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.89)
• Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samples (NeurIPS 2025 · similarity 0.88)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.88)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.88)
• MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.87)
• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.87)