Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes qGOFT, a parameter-efficient orthogonal fine-tuning method that composes O(d) Givens rotations (with a parallel O(log d) sparse scheme) and relaxes them with learnable norms and angles under a soft orthogonality regularizer.
Problem
Orthogonal fine-tuning (OFT) needs O(d^2) trainable parameters for full orthogonal matrices, and block-diagonal approximations sacrifice expressiveness; strict orthogonality also prevents adapting slight semantic shifts in downstream tasks.
Approach
Show that d-1 Givens rotations can map any vector to any point on the same sphere, so SO(d) transforms can be parameterized with O(d) angles (GOFT) and arranged in a parallel O(log d) sparse product; qGOFT then relaxes each Givens rotation into a general 2x2 quasi-Givens matrix with a soft orthogonality penalty, allowing adjustable norms and relative angles.
Key results
- On GLUE with DeBERTaV3-base, qGOFT (0.33M params) reaches 89.35 average versus 88.78 for OFT b=8 (0.66M) and 87.80 for OFT b=4.
- On VTAB-1K with ViT-B/16, qGOFT achieves 79.4 average accuracy versus 78.0 for OFT b=8 (0.147M) with 0.073M parameters.
- On MMLU 5-shot with LLaMA2-7B, GOFT (0.007% params) gets 46.36 average versus 45.93 for OFT b=16 (0.13% params).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GOFT rotations with quasi-Givens rotations that allow norm and angular adjustment | GOFT on GLUE · GLUE development set | All average: 88.78 -> 89.35 | Helped |
| replace GOFT rotations with quasi-Givens rotations that allow norm and angular adjustment | GOFT on SQuAD v1.1 · SQuAD v1.1 | EM: 86.1 -> 87.1 | Helped |
| replace GOFT rotations with quasi-Givens rotations that allow norm and angular adjustment | GOFT on SQuAD v1.1 · SQuAD v1.1 | F1: 92.7 -> 93.2 | Helped |
| replace GOFT rotations with quasi-Givens rotations that allow norm and angular adjustment | GOFT on LLaMA2-7B instruction tuning · MMLU 5-shot | average accuracy: 46.36 -> 46.12 | Hurt |
| replace GOFT rotations with quasi-Givens rotations that allow norm and angular adjustment | GOFT on LLaMA2-7B instruction tuning · Vicuna-Eval | GPT-4-turbo score: 5.96 -> 5.57 | Hurt |
| add norm adjustment only (ablation) | GOFT on QNLI and RTE · QNLI and RTE | performance | Helped |
| add angular adjustment as well as norm adjustment (ablation) | GOFT* on QNLI and RTE · QNLI and RTE | performance | Helped |
| increase orthogonality regularization strength to 0.5 (ablation) | qGOFT · QNLI and RTE | performance | Hurt |
| set orthogonality regularization strength to 0.001 rather than 0.05 (ablation) | qGOFT · QNLI | performance | Helped |
| set orthogonality regularization strength to 0.05 rather than 0.001 (ablation) | qGOFT · RTE | accuracy | Helped |
Surprising findings
- GOFT outperforms qGOFT on LLaMA2-7B instruction tuning while qGOFT is better on GLUE, which the authors attribute to the smaller pretrain-finetune gap for decoder models and larger model scale.
- Training runtime of (q)GOFT is about 3.3-3.5x that of OFT due to the sparse Givens products.
Limitations
- Training runtime overhead is about 3.3-3.5x that of OFT due to O(log d) sparse matrix multiplications.
- The regularization strength lambda must be tuned per task via trial and error.
- Results for qGOFT vs GOFT are task-dependent, with GOFT better on LLaMA instruction tuning.
Experimental setup
Datasets: GLUE SQuAD v1.1 Alpaca-cleaned-51K VTAB-1K ImageNet-21K
Benchmarks: GLUE MMLU Vicuna-Eval SQuAD v1.1 VTAB-1K
Metrics: accuracy Matthews correlation Pearson correlation exact match F1 GPT-4-turbo score top-1 accuracy
Baselines: Full Fine-tuning BitFit H-Adapter P-Adapter LoRA AdaLoRA FacT OFT
Builds on: OFT LoRA Givens rotation Cayley parameterization
Closest papers by meaning
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.89)
- Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (ICLR 2024 · similarity 0.87)
- CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation (NeurIPS 2026 · similarity 0.87)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.86)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.