Research map / Architectures and efficiency / Pruning and efficient tuning
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TopLoRA, which generates a token-dependent diagonal matrix from each input token to modulate LoRA's shared input-output projection (B Σ_X A), improving accuracy over LoRA at the same rank on GLUE and LLM reasoning benchmarks.
Problem
Standard LoRA applies the same input-output projection to all tokens, limiting its ability to capture token-specific information; the role of rank in expressiveness is also unclear.
Approach
Decompose LoRA weights into input space, output space, and input-output projection P=R_B L_A, then make P token-dependent by inserting a diagonal matrix Σ_X produced from each input token via a learned projector (Kaiming init, RMSNorm, exponential), giving token-wise weights B Σ_X A without increasing rank.
Key results
- On GLUE with RoBERTa-Base, TopLoRA (r=8) averages 84.14 vs. 82.55 for LoRA (r=8) and 83.60 for LoRA (r=32); TopLoRA (r=16) reaches 84.48.
- On math reasoning with Gemma-7B, TopLoRA (r=8) averages 73.11 vs. LoRA (r=8) 71.44, and with LLaMA-3-8B 71.84 vs. 68.19.
- On commonsense reasoning with LLaMA-3-8B, TopLoRA (r=16) averages 86.84 vs. LoRA (r=32) 86.72, with fewer parameters than LoRA r=32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token-wise input-output projections at rank 8 instead of LoRA rank 32 | RoBERTa-Base · GLUE | Avg accuracy: 83.60 -> 84.14 | Helped |
| Add token-wise input-output projections at rank 16 instead of LoRA rank 32 | RoBERTa-Large · GLUE | Avg accuracy: 86.35 -> 87.55 | Helped |
| Add token-wise input-output projections | Gemma-7B · Mathematical reasoning | Avg accuracy: 72.85 -> 73.28 | Helped |
| Add token-wise input-output projections | LLaMA-3-8B · Mathematical reasoning | Avg accuracy: 71.67 -> 72.95 | Helped |
| Add token-wise input-output projections | Qwen2.5-14B · Mathematical reasoning | Avg accuracy: 80.16 -> 80.76 | Helped |
| Add token-wise input-output projections | Gemma-7B · Commonsense reasoning | Avg accuracy: 87.01 -> 88.44 | Helped |
| Add token-wise input-output projections | LLaMA-3-8B · Commonsense reasoning | Avg accuracy: 86.72 -> 86.84 | Helped |
| Add token-wise input-output projections | Qwen2.5-14B · Commonsense reasoning | Avg accuracy: 91.73 -> 91.93 | Helped |
| Add exponential transformation to the projector (ablation) | TopLoRA on Gemma-7B · Mathematical reasoning | Avg accuracy: 72.84 -> 73.11 | Helped |
| Add RMSNorm to the projector (ablation) | TopLoRA on Gemma-7B · Mathematical reasoning | Avg accuracy: 72.69 -> 73.11 | Helped |
| Add exponential transformation to the projector (ablation) | TopLoRA on LLaMA-3-8B · Mathematical reasoning | Avg accuracy: 70.56 -> 71.84 | Helped |
| Add RMSNorm to the projector (ablation) | TopLoRA on LLaMA-3-8B · Mathematical reasoning | Avg accuracy: 71.77 -> 71.84 | No clear effect |
Surprising findings
- Authors note that TopLoRA gains at rank 8 exceed those from quadrupling LoRA's rank on math reasoning, suggesting token-wise projection is more effective than rank increase.
Limitations
- Token-wise weights cannot be merged into pretrained weights after fine-tuning, adding inference latency.
- Evaluated only on language tasks; applicability to vision tasks is untested.
- Lacks theoretical justification for the advantages of token-wise projections.
Experimental setup
Datasets: GLUE math reasoning training set (10K) commonsense reasoning training set (170K)
Benchmarks: GLUE AddSub MultiArith SingleEq GSM8K AQuA SVAMP BoolQ PIQA SIQA HellaSwag WinoGrande
Metrics: accuracy Matthew's correlation Pearson correlation
Baselines: LoRA DoRA MELoRA HydraLoRA
Builds on: LoRA DoRA MELoRA HydraLoRA MoELoRA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.94)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.93)
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.93)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.93)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.93)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.