Research map / Architectures and efficiency / Pruning and efficient tuning
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Orthogonal Butterfly (BOFT), a parameter-efficient orthogonal finetuning method that parameterizes dense orthogonal matrices as products of sparse butterfly-factorized orthogonal matrices, and evaluates it on language, vision, and diffusion adaptation tasks.
Problem
Orthogonal finetuning (OFT) requires many trainable parameters due to high-dimensional orthogonal matrices, and its block-diagonal sparsity compromises expressivity and lacks principled sparsity patterns.
Approach
Compose a dense orthogonal matrix from a product of O(log d) sparse orthogonal butterfly components (inspired by the Cooley-Tukey FFT), giving O(d log d) parameters, and subsume OFT as the single-factor case; also adds multiplicative dropout and identity initialization.
Key results
- On GLUE with DeBERTaV3-base, BOFT (b=8, m=2) reaches 89.89 average with 0.75M parameters, versus 89.77 for OFT (b=16, 0.79M) and 88.50 for LoRA (r=8, 1.33M).
- On VTAB-1K with DINOv2-large, BOFT (m=4, b=4) averages 77.4% and BOFT (m=2, b=8) averages 77.9%, versus 76.6% for LoRA (r=4) and 77.3% for OFT.
- On GSM8K and MATH with Llama-2-7B, BOFT reaches 50.6 and 8.6 versus 50.2 and 7.8 for LoRA (r=32), using about half of LoRA's parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace block-diagonal OFT with butterfly factorization | OFT on GLUE · GLUE development set | All: 89.77 -> 89.89 | Helped |
| replace block-diagonal OFT with butterfly factorization | OFT on MMLU · MMLU 5-shot | Avg. accuracy (%): 47.5 -> 47.9 | Helped |
| replace block-diagonal OFT with butterfly factorization | OFT on MMLU · MMLU 0-shot | Avg. accuracy (%): 46.2 -> 46.7 | Helped |
| replace LoRA with BOFT | Llama-2-7B math question answering · GSM8K | accuracy: 50.2 -> 50.6 | Helped |
| replace LoRA with BOFT | Llama-2-7B math question answering · MATH | accuracy: 7.8 -> 8.6 | Helped |
| replace OFT with BOFT b=8, m=2 | DINOv2-large transfer learning · VTAB-1K | Average accuracy (%): 77.3 -> 77.9 | Helped |
| use BOFT b=4, m=4 instead of b=8, m=2 (ablation) | DINOv2-large transfer learning · VTAB-1K | Average accuracy (%): 77.9 -> 77.4 | Hurt |
| use BOFT b=2, m=6 instead of b=8, m=2 (ablation) | DINOv2-large transfer learning · VTAB-1K | Average accuracy (%): 77.9 -> 77.1 | Hurt |
| replace HQ-SAM with BOFT-SAM | SAM segmentation adaptation · HQSeg-44K | Average mIoU: 89.1 -> 89.5 | Helped |
| replace OFT with BOFT r=8, m=4 | Stable Diffusion landmark control · Landmark-to-Face | Error: 6.407 -> 5.667 | Helped |
| increase the number of butterfly components m (ablation) | BOFT with r=32 on Stable Diffusion · Controllable generation | control performance | Helped |
| increase the number of butterfly components m (ablation) | BOFT with r=32 on Stable Diffusion · Controllable generation | convergence speed | Helped |
Surprising findings
- Setting butterfly components to identity after training interpolates smoothly from the controlled model back to plain Stable Diffusion without retraining.
- Full finetuning on VTAB-1K (74.6 average) is outperformed by several PEFT methods, and the paper notes that greater expressiveness does not necessarily yield better finetuning performance.
Limitations
- Training runtime overhead is slightly larger than OFT due to multiplying several orthogonal factors.
- Unclear whether the butterfly network is the most efficient information-transmission topology.
Experimental setup
Datasets: GLUE VTAB-1K Alpaca MetaMathQA-40K HQSeg-44K ADE20K CelebV-HQ DreamBooth
Benchmarks: GLUE MMLU GSM8K MATH VTAB-1K HQSeg-44K ControlNet landmark-to-face error
Metrics: accuracy Matthews correlation mIoU mBIoU landmark error Top-1 accuracy
Baselines: Full Finetuning BitFit AdaLoRA LoRA OFT GLoRA FacTT HQ-SAM H-Adapter P-Adapter
Builds on: Orthogonal Finetuning (OFT) LoRA Cayley parameterization HQ-SAM Cooley-Tukey FFT butterfly structure
Closest papers by meaning
- Group and Shuffle: Efficient Structured Orthogonal Parametrization (NeurIPS 2024 · similarity 0.89)
- Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation (ICML 2024 · similarity 0.87)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.87)
- FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models (NeurIPS 2026 · similarity 0.86)
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform (ICML 2024 · similarity 0.85)
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.