Research map / Architectures and efficiency / Attention and transformers
OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OrthoRank, a training-free dynamic token selection method that updates only the tokens most orthogonal to the attention-sink token in selected layers, combined with layer pruning, to outperform SLEB and Shortened LLaMA at equal sparsity.
Problem
Layer pruning applies a static decision to all tokens and cannot adapt computation per token, while existing token-level dynamic methods (early exit, Mixture-of-Depths) require extra routers or training.
Approach
Uses the cosine similarity between normalized hidden states of the attention-sink token and other tokens as a signal: tokens that are more orthogonal to the nearly static sink token are the ones that move most toward it, so they are selected for full computation in selected layers, while unselected tokens skip query/FFN computation but still provide keys and values. No router or fine-tuning is nee
Key results
- At 20% sparsity on Llama-2-7B, OrthoRank reaches C4 perplexity 10.04 versus 10.90 for SLEB (dense 7.26).
- On LongBench with Llama-3-8B at 20% sparsity (2048 calibration), OrthoRank scores 25.23 versus 17.85 for SLEB.
- OrthoRank gives 1.18x throughput improvement at 20% sparsity on Llama-2-13B, while attention-based selection gives 0.71x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OrthoRank token selection to SLEB at 20% sparsity | SLEB on C4 validation · C4 validation, Llama-2-13B | perplexity: 9.42 -> 8.74 | Helped |
| Add OrthoRank token selection to SLEB at 20% sparsity | SLEB on zero-shot tasks · Zero-shot tasks, Llama-2-13B | mean accuracy (%): 62.97 -> 66.99 | Helped |
| Add OrthoRank token selection to SLEB | SLEB on LongBench · LongBench, Llama-3-8B, 2048 calibration context and 10% sparsity | average accuracy: 25.10 -> 29.61 | Helped |
| Use OrthoRank token selection with Shortened LLaMa at 10% sparsity | Shortened LLaMa on C4 validation · C4 validation, Llama-2-13B | perplexity: 7.93 -> 7.60 | Helped |
| Select tokens with larger sink inner product instead of more orthogonal tokens (ablation) | OrthoRank token selection · Token selection ablation, Llama-2-13B | perplexity: 8.74 -> 11.85 | Hurt |
| Use unnormalized hidden states for token selection (ablation) | OrthoRank · Similarity measurement ablation, Llama-2-13B | perplexity: 8.74 -> 9.64 | Hurt |
| Skip KV calculations for unselected tokens (ablation) | OrthoRank · Unselected-token KV ablation, Llama-2-13B | perplexity: 8.74 -> 9.77 | Hurt |
| Increase token selection ratio with layer depth (ablation) | OrthoRank at 20% overall sparsity · Token ratio assignment, Llama-2-13B | perplexity: 8.74 -> 9.10 | Hurt |
| Decrease token selection ratio with layer depth (ablation) | OrthoRank at 20% overall sparsity · Token ratio assignment, Llama-2-13B | accuracy: 66.99 -> 68.18 | Helped |
| Add OrthoRank token selection to SLEB | SLEB on TruthfulQA · TruthfulQA, Llama-2-13B | generation BLEU: 20.82 -> 23.6 | Helped |
Surprising findings
- Attention-based token selection achieves competitive perplexity but lower zero-shot accuracy and substantially reduced throughput.
- Skipping KV computation for unselected tokens significantly degrades performance, so unselected tokens still need KV calculation.
Limitations
- Selection is restricted to the first token as sink; other delimiter tokens acting as sinks are not used.
- Sparsity above roughly 40-50% causes sharp perplexity increases.
- Some zero-shot tasks and LongBench subsets show SLEB outperforming OrthoRank.
Experimental setup
Datasets: WikiText-2 C4 LongBench
Benchmarks: C4 perplexity WikiText-2 perplexity PIQA WinoGrande HellaSwag ARC-Challenge ARC-Easy LongBench TruthfulQA
Metrics: perplexity accuracy BLEU MC1 throughput
Baselines: SLEB Shortened LLaMA Random token selection Attention-based token selection Mixture-of-Depths
Builds on: SLEB Shortened LLaMA attention sink
Closest papers by meaning
- Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing (ICML 2026 · similarity 0.88)
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.88)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.88)
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.88)
- Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.88)
- L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.