Research map / Large language models / Language, safety and interpretability

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LowRankArena, a standardized evaluation platform and checkpoint zoo for SVD-based LLM compression, re-auditing five SVD methods under aligned budgets, tasks, and vLLM inference.

Problem

Reported gains of SVD-based low-rank LLM compression methods are hard to compare because prior evaluations use different benchmarks, task versions, keep-ratio definitions, backbones, and inference setups, and often bundle quantization or remapping into the budget.

Approach

A unified evaluation protocol with a uniform-precision keep-ratio budget, aligned task versions, separated comparison regimes (uniform-precision SVD as primary), and end-to-end vLLM inference measurement, plus release of over 3 TiB of compressed checkpoints; the audit shows rankings are conditional on backbone, keep ratio, and metric.

Key results

• On Llama-3.1-8B at 60% keep ratio, MoDeGPT keeps C4 perplexity at 23.40 vs 1187.78 for SVD-LLM v1 and 461.21 for Basis Sharing, with MCQ average 0.504 vs 0.360 and 0.396.
• Method rankings shift across Llama-1, Llama-3.1, and Qwen3 backbones; e.g., ASVD drops from #2 to #5 at 80% keep moving from Llama-1 to Llama-3.1.
• In the A100 serving audit, SVD methods give up to 4.20x TTFT speedup in prefill-heavy workloads, but decode-heavy throughput drops to 0.80x for SVD-LLM and E2E speedup reaches only 0.68x-1.06x in fixed-setting audit.

Surprising findings

• Multiple-choice accuracy can remain near-competitive while C4 perplexity degrades by orders of magnitude, e.g., SVD-LLM at 60% keep reaches 1187.78 C4 PPL.
• Nominal low-rank savings yield speedups concentrated in prefill; several decode-heavy settings are slower than dense (E2E 0.68x for SVD-LLM on RTX A5000).

Limitations

• DoBi-SVD omitted from calibration audit because repeated runs did not complete within compute budget.
• Several released 70B codebases fail to produce artifacts under a one-H200 recipe, limiting large-model coverage.
• Instruction-tuned results treated as auxiliary due to sensitivity to prompt templates and decoding.

Experimental setup

Datasets: WikiText-2 C4 Pile-Val BoolQ ARC-Easy ARC-Challenge WinoGrande PIQA HellaSwag OpenBookQA MathQA MMLU-Math

Benchmarks: WikiText-2 PPL C4 PPL BoolQ ARC-Easy ARC-Challenge WinoGrande PIQA HellaSwag OpenBookQA MathQA MMLU-Math MMLU-Pro

Metrics: perplexity accuracy (acc_norm) macro average quality retention TTFT E2E latency input/output throughput

Baselines: ASVD SVD-LLM DoBi-SVD Basis Sharing MoDeGPT LLM-Pruner SliceGPT

Builds on: ASVD SVD-LLM DoBi-SVD Basis Sharing MoDeGPT

Closest papers by meaning

• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.93)
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.91)
• Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.91)
• Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.90)
• SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.90)
• Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression (ICLR 2025 · similarity 0.90)