Research map / Architectures and efficiency / Pruning and efficient tuning

CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CORA, a per-slice coherent orthogonal rotation adapter applied to the rank-r SVD reconstruction of each pretrained weight, using a Cayley-parameterized block rotation plus a shared diagonal spectrum scale, with about 8x fewer parameters than LoRA-family baselines on commonsense, code, and math tasks.

Problem

Existing SVD-based PEFT methods (PiSSA, MiLoRA) and orthogonal finetuning methods only partially realize the coherent rotation-spectrum form of minimum-perturbation finetuning, and the global form does not specify how to realize it efficiently at the row-slice granularity of practical adapters.

Approach

Extends the coherent rotation theorem to row slices of the rank-r SVD reconstruction and parameterizes each slice's rotation as Q_i = U_i^T R_i U_i with a single orthogonal R_i (Cayley map of a skew-symmetric matrix), plus an optional shared per-layer diagonal spectrum scale, so left and right bases rotate coherently by construction.

Key results

• CORA r=16 reaches 82.16% commonsense 8-task average on LLaMA-2-7B at 6.6M parameters, above DoRA r=32 (79.7%) and MiLoRA r=32 (79.2%) at 56M parameters.
• On LLaMA-3-8B, CORA r=16 at 10.3M parameters reaches 86.65% commonsense average, versus 81.9% for MiLoRA r=32.
• On HumanEval, CORA r=16 reaches 48.2 Pass@1 on LLaMA-3-8B at 10.3M parameters, above the DiaBlo N=64 (43.2) reference at 121M.

Surprising findings

• On MATH, the principal-subspace CORA variant (12.7) trails MiLoRA r=64 (17.8), which the authors attribute to spectral locality of mathematical knowledge.
• Dropping the V-side rotation (U-only variant) reduces HumanEval Pass@1 on Mistral-7B by 2.4 to 9.8 points, indicating two-sided coherence matters in practice.

Limitations

• Requires an offline SVD of every adapted weight and caching of per-slice factors before training.
• Slice-wise forward adds matmuls per layer relative to LoRA, and largest evaluated backbone is 8B.
• Evaluation limited to instruction-tuned commonsense, math, and code; vision, multilingual, long-context, and continual learning unexplored.

Experimental setup

Datasets: commonsense_170k MetaMathQA-395K CodeAlpaca

Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-Easy ARC-Challenge OpenBookQA HumanEval GSM8K MATH

Metrics: accuracy Pass@1 exact match

Baselines: LoRA DoRA PiSSA MiLoRA LoRI DiaBlo Full FT

Builds on: LoRA PiSSA MiLoRA OFT DoRA

Closest papers by meaning

• Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation (ICML 2024 · similarity 0.87)
• LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.86)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.86)
• SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.85)
• PoLAR: Polar-Decomposed Low-Rank Adapter Representation (NeurIPS 2025 · similarity 0.85)
• Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation (NeurIPS 2024 · similarity 0.85)