Research map / Architectures and efficiency / Attention and transformers
Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Rank Allocation and the RaNA adapter, which applies input-dependent rank masking to SVD-based low-rank decompositions of linear layers to speed up Llama2-7b, Gemma-2b and Pythia inference without relying on activation sparsity.
Problem
Neuron-adaptive methods for accelerating Transformer inference rely on activation sparsity, do not apply to attention layers, and use costly neuron maskers, so they degrade on modern SwiGLU-based LLMs.
Approach
Generalize neuron adapters to rank adapters: decompose each linear layer as A(r(x) ⊙ Bx) with A and B from the SVD of WX (calibration activations), and use an input-dependent B-masker that keeps ranks with large (Bx)^2, applied to QKV, up and gate projections, with neuron thresholding for down projections.
Key results
- On Llama2-7b at 42% FLOP compression, RaNA reaches 56.37% avg accuracy and 8.04 PPL versus CATS at 47.51% and 12.36 PPL.
- On Gemma-2b at 44% FLOP compression, RaNA reaches 48.16% avg accuracy and 13.83 PPL versus CATS at 43.01% and 21.02 PPL (CATS at 47% compression).
- RaNA's average MLP reconstruction error on Llama2-7b is 10.5% versus 17.22% for CATS at ~50% FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RaNA instead of CATS at 42% FLOP compression | Llama2-7b · Llama2-7b, 42% FLOP compression | average accuracy: 47.51% -> 56.37% | Helped |
| use RaNA instead of CATS at 42% FLOP compression | Llama2-7b · Llama2-7b, 42% FLOP compression | perplexity: 12.36 -> 8.04 | Helped |
| use RaNA instead of SliceGPT at 42% FLOP compression | Llama2-7b · Llama2-7b, 42% FLOP compression | average accuracy: 43.21% -> 56.37% | Helped |
| use RaNA instead of SliceGPT at 42% FLOP compression | Llama2-7b · Llama2-7b, 42% FLOP compression | perplexity: 13.69 -> 8.04 | Helped |
| use RaNA instead of CATS at approximately 44% model FLOP compression | Gemma-2b · Gemma-2b, high compression | perplexity: 7 less perplexity-points | Helped |
| compress QKV layers in addition to MLP layers (ablation) | RaNA · Llama2-7b, ~31% FLOP compression, no fine-tuning | perplexity: 8.79 -> 8.40 | Helped |
| add MLP-level FLOP allocation (ablation) | RaNA compressing MLP and QKV layers · Llama2-7b, ~31% FLOP compression, no fine-tuning | perplexity: 9.10 -> 8.40 | Helped |
Surprising findings
- Static SliceGPT lags far behind both adaptive methods at matched FLOP rates, and at 15-20% compression CATS matches or slightly beats RaNA on Llama2-7b and Gemma-2b accuracy.
Limitations
- Naive PyTorch implementation gives no practical speedup, requiring a custom Triton kernel.
- Gemma-2b QKV layers are not adapted, and the FLOP allocation is greedy/grid-searched per layer rather than model-level.
- At low compression (15-20%), RaNA does not consistently beat CATS.
Experimental setup
Datasets: RedPajama The Pile HellaSwag PIQA WinoGrande ARC-Easy ARC-Challenge RACE
Benchmarks: ARC-Easy ARC-Challenge HellaSwag PIQA RACE WinoGrande RedPajama perplexity Pile perplexity
Metrics: accuracy perplexity normalized reconstruction error FLOP compression rate latency
Baselines: CATS SliceGPT Neuron-Adaptive LLRA (linear layer rank adapters with MLP maskers)
Builds on: CATS SliceGPT neuron adapters
Closest papers by meaning
- Transformer-Squared: Self-adaptive LLMs (ICLR 2025 · similarity 0.86)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.86)
- Sparse High Rank Adapters (NeurIPS 2024 · similarity 0.85)
- Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference (AAAI 2025 · similarity 0.85)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.85)
- On the Duality between Gradient Transformations and Adapters (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.