Research map / Architectures and efficiency / Pruning and efficient tuning
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Householder Reflection Adaptation (HRA), which fine-tunes frozen weights by multiplying them with a chain of learnable Householder reflections, unifying LoRA-style and orthogonal fine-tuning, with an orthogonality regularizer trading capacity for regularity.
Problem
Parameter-efficient fine-tuning currently splits into low-rank adaptation (LoRA) and orthogonal fine-tuning (OFT/BOFT), with no unified framework; these methods trade off parameter count, computation, and retention of pre-training knowledge differently.
Approach
Multiply each frozen weight matrix by an orthogonal matrix built from a chain of r learnable Householder reflections. This is equivalent to an adaptive LoRA with an orthogonal guarantee. Regularizing the orthogonality of the reflection vectors (via lambda, or strict Gram-Schmidt orthogonalization) controls model capacity versus regularity.
Key results
- On GLUE, HRA r=8 with lambda=0 reaches 90.10 average with 0.66M parameters, versus 89.89 for BOFT (0.75M) and 89.77 for OFT (0.79M).
- On LLaMA2-7B mathematical reasoning, HRA r=16 uses half of BOFT's trainable parameters yet surpasses its performance, per the paper's text.
- In controllable generation, HRA reaches the smallest parameter count (0.89M) and best L2F error (5.46) among compared methods on Stable Diffusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HRA r=8 with no orthogonality regularization rather than BOFT | DeBERTaV3-base · GLUE development set | average score (%): 89.89 -> 90.10 | Helped |
| Add strict orthogonality via Gram-Schmidt (ablation) | HRA r=8 · GLUE development set | average score (%): 90.10 -> 89.13 | Hurt |
| Set orthogonality weight to 1e-4 rather than 1e-1 (ablation) | HRA r=32 · GSM8K validation | accuracy (%): 53.6 -> 56.3 | Helped |
| Add orthogonality regularization with λ=1e-4 (ablation) | HRA r=32 · MATH validation | accuracy (%): 9.0 -> 9.3 | Helped |
| Enforce strict orthogonality via Gram-Schmidt (ablation) | HRA r=32 · GSM8K validation | accuracy (%): 56.3 -> 52.8 | Hurt |
| Use HRA rather than LoRA for MATHQA adaptation | LLaMA2-7B · Classic NLP tasks after MATHQA fine-tuning | overall impact (%): -6.03% -> -1.79% | Helped |
| Remove orthogonality regularization (ablation) | HRA · Subject-driven generation | DINO: 0.661 -> 0.670 | Helped |
| Enforce strict orthogonality via Gram-Schmidt (ablation) | HRA · Subject-driven generation | CLIP-T: 0.255 -> 0.274 | Helped |
| Enforce strict orthogonality via Gram-Schmidt (ablation) | HRA · C2I (COCO) | IoU: 0.213 -> 0.201 | Hurt |
| Use HRA rather than LoRA | LLaMA2-7B · MetaMathQA adaptation | training time (hours): 45 -> 30 | Helped |
| Use HRA rather than LoRA | LLaMA2-7B · MetaMathQA adaptation | peak memory usage (GB): 279 -> 287 | Hurt |
Surprising findings
- Strict orthogonality (lambda=infinity, Gram-Schmidt) hurts performance on GLUE and controllable generation, suggesting these tasks need adapter capacity beyond orthogonal constraints.
- HRA with no orthogonality regularization (lambda=0) performs as well as or better than regularized variants on several tasks.
Limitations
- Peak memory usage is slightly higher than LoRA and OFT in the efficiency table (287 GB vs 279 GB and 282 GB).
- Theoretical analysis of how the regularization weight lambda affects behavior is deferred to future work.
- Evaluation of LLM adaptation is limited to LLaMA2-7B and GLUE with DeBERTaV3-base.
Experimental setup
Datasets: GLUE MetaMathQA DreamBooth subjects COCO CelebA-HQ ADE20K
Benchmarks: GLUE GSM8K MATH ARC HellaSwag MMLU Winogrande HumanEval DINO CLIP-I CLIP-T LPIPS
Metrics: accuracy Matthews correlation average correlation DINO CLIP-I CLIP-T LPIPS IoU F1 mIoU
Baselines: Full Fine-tune BitFit H-Adapter P-Adapter LoRA AdaLoRA OFT BOFT DreamBooth ControlNet T2I-Adapter COFT
Builds on: LoRA OFT BOFT AdaLoRA DoRA
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.89)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.87)
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.87)
- Decoupling Angles and Strength in Low-rank Adaptation (ICLR 2025 · similarity 0.87)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.87)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.