Research map / Large language models / Instruction and fine-tuning
3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoAd, a PEFT method that applies trainable 2D rotations (with scaling) to pairs of hidden dimensions, enabling element-wise batched multi-adapter serving and composability via distributed interchange intervention.
Problem
PEFT adapters are costly to serve in heterogeneous batches (batch matrix multiplication overhead), and PEFT methods offer limited interpretability/composability of learned subspaces.
Approach
Finetuning mainly changes angular rather than magnitude of representations, so RoAd rotates adjacent dimension pairs by trainable angles with trainable scaling; the block-sparse rotation is reformulated as two element-wise products, giving batching overhead comparable to (IA)^3 and orthogonal subspaces usable for composition.
Key results
- On GLUE with RoBERTa-base, RoAd1 (0.07% params) reaches 85.6 average, matching full finetuning (85.6) and outperforming LoReFT (84.2).
- On commonsense reasoning with LLaMA-7B, RoAd2/RoAd4 reach 80.2 average vs LoReFT 80.2 and LoRA 74.7, with <0.1% trainable parameters.
- RoAd achieves about twice the throughput of LoRA in heterogeneous batched serving; RoAd4 with 0.08% params reaches 96.9% of LoRA (4.61% params) accuracy on LLaVA GQA/SQA/VQAT/POPE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoAd1 rather than LoReFT | LLaMA-7B · Eight commonsense reasoning tasks | Avg. accuracy: 80.2 vs. 79.2 | No clear effect |
| Increase RoAd1 parameterization to RoAd2 (ablation) | RoAd on LLaMA-7B · Eight commonsense reasoning tasks | Avg. accuracy: 79.2 -> 80.2 | Helped |
| Increase RoAd1 parameterization to RoAd4 (ablation) | RoAd on LLaMA-7B · Four arithmetic reasoning tasks | Avg. accuracy: 44.0 -> 45.8 | Helped |
| Increase RoAd1 parameterization to RoAd2 (ablation) | RoAd on LLaMA-7B · Four arithmetic reasoning tasks | Avg. accuracy: 44.0 -> 46.2 | Helped |
| Increase RoAd1 parameterization to RoAd4 (ablation) | RoAd on LLaMA-13B · Four arithmetic reasoning tasks | Avg. accuracy: 51.9 -> 52.3 | Helped |
| Apply RoAd1 only to the first feed-forward layer (ablation) | RoBERTa-base on GLUE · GLUE dev | Avg.: 85.6 -> 85.1 | Hurt |
| Combine RoAd1 with LoRA instead of using LoRA alone | LLaVA1.5-7B · Visual instruction tuning | Avg.: 68.5 -> 68.5 | No clear effect |
| Use RoAd1 instead of OFT n=2048 | LLaMA-7B finetuning · 100 iterations | Training time (s): 1249 -> 25 | Helped |
| Use RoAd1 instead of OFT n=2048 | LLaMA-7B finetuning · 100 iterations | Peak GPU memory (GB): 40 -> 23 | Helped |
| Use RoAd1 rather than LoReFT | LLaMA-13B · Eight commonsense reasoning tasks | Avg. accuracy: 83.3 vs. 83.0 | Hurt |
Surprising findings
- RoAd1 with 0.07% trainable parameters matches or outperforms full finetuning on GLUE average with RoBERTa-base.
- Pilot study: changing the angle of representations matters more than magnitude; magnitude-only classifiers underperform even a random backbone.
Limitations
- Scalability: number of trainable parameters cannot be indefinitely increased with RoAd; needs combination with LoRA for knowledge-intensive tasks.
- On arithmetic reasoning with LLaMA-7B, RoAd is worse than LoRA and Adapter-P despite using far fewer parameters.
- Composability experiments are qualitative and small (1K samples, LLaMA-7B only).
Experimental setup
Datasets: GLUE Math10K BoolQ PIQA SIQA HellaSwag WinoGrande ARC OBQA AQuA GSM8K MAWPS
Benchmarks: GLUE BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA AQuA GSM8K MAWPS
Metrics: accuracy Matthews correlation Pearson correlation win rate throughput peak GPU memory training time
Baselines: LoRA Adapter BOFT OFT BitFit (IA)^3 RED LoReFT Prefix-tuning DoRA FLoRA
Builds on: LoRA OFT BOFT LoReFT RoPE
Closest papers by meaning
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.87)
- D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · similarity 0.87)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.87)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.86)
- Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.