Research map / Image generation / Text-to-image diffusion
Group and Shuffle: Efficient Structured Orthogonal Parametrization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Group-and-Shuffle (GS) structured matrices that generalize Monarch and block-butterfly classes, and uses orthogonal GS parametrization for parameter-efficient fine-tuning (GSOFT, Double GSOFT) and 1-Lipschitz orthogonal convolutions.
Problem
Parametrizing dense orthogonal matrices efficiently and expressively for orthogonal fine-tuning and orthogonal convolutions, where block-diagonal (OFT) is too restrictive and butterfly products (BOFT) need many matrices and are slow.
Approach
Define GS-matrices as alternating products of block-diagonal matrices and fixed permutations, prove that enforcing orthogonality blockwise is sufficient for orthogonal GS matrices, and show two GS factors with shuffle permutation P_(k,br) already yield dense matrices, reducing matrix count from 1+ceil(log2 r) to 1+ceil(log_b r).
Key results
- On GLUE with RoBERTa-base, GSOFT (b=8, 1.42M params) reaches 86.67 average versus 86.53 for LoRA (r=8, 1.33M) and 85.84 for BOFT (b=8, m=2, 1.42M).
- On DreamBooth subject-driven generation with Stable Diffusion 2 base, GSOFT attains CLIP-T 0.256 versus 0.244 for BOFT at comparable parameter count, and Double GSOFT reaches CLIP-I 0.815 with 6.5M parameters.
- On LipConvnet-15 / CIFAR-100, GS-SOC with (4, -) groups cuts parameters from 24.1M to 6.81M with 1.64x speedup while matching accuracy (43.48% vs 43.15%) and robust accuracy (29.26% vs 29.18%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace OFT, BOFT, or LoRA with GSOFT b=8 | RoBERTa-base fine-tuning · GLUE | ALL: 86.53 -> 86.67; 86.27 -> 86.67; 85.84 -> 86.67 | Helped |
| use Double GSOFT r=64 instead of LoRA rank 32 | Stable Diffusion-2-base subject-driven generation · DreamBooth | CLIP-I: 0.819 -> 0.815 | Hurt |
| use GSOFT r=32 instead of LoRA rank 4 | Stable Diffusion-2-base subject-driven generation · DreamBooth | CLIP-T: 0.246 -> 0.256 | Helped |
| replace SOC with GS-SOC groups (4, -) | LipConvnet-15 · CIFAR-100 | Speedup: 1 -> 1.64 | Helped |
| use paired instead of unpaired channel permutation with MaxMinPermuted (ablation) | GS-SOC groups (4, -) · CIFAR-100 | Accuracy: 40.46% -> 43.48% | Helped |
| use paired instead of unpaired channel permutation with MaxMinPermuted (ablation) | GS-SOC groups (4, -) · CIFAR-100 | Robust Accuracy: 26.18% -> 29.26% | Helped |
| use MaxMinPermuted instead of MaxMin with paired permutation (ablation) | GS-SOC groups (4, -) · CIFAR-100 | Accuracy: 37.99% -> 43.48% | Helped |
| use MaxMinPermuted instead of MaxMin with paired permutation (ablation) | GS-SOC groups (4, -) · CIFAR-100 | Robust Accuracy: 24.19% -> 29.26% | Helped |
| add a second grouped exponential convolution with kernel size 1 (ablation) | GS-SOC groups (4, -) · CIFAR-100 | Accuracy: 43.48% -> 43.42% | No clear effect |
| vary the second convolution groups from 1 to 2 or 4 (ablation) | GS-SOC · CIFAR-100 | Accuracy: 43.42% -> 42.86%; 43.42% -> 42.75% | Hurt |
| vary the second convolution groups from 1 to 2 or 4 (ablation) | GS-SOC · CIFAR-100 | Robust Accuracy: 29.56% -> 28.98%; 29.56% -> 28.7% | Mixed |
Surprising findings
- Monarch matrices cannot represent the square-block orthogonal structure needed under low parameter budgets because their block-count constraint couples k_L and k_R.
- Using MaxMin activation with standard channel shuffling degraded GS-SOC accuracy; the paired permutation with MaxMinPermuted was needed to recover performance.
Limitations
- GSOFT training is still slower than LoRA.
- The parametrization trades expressivity for parameter efficiency and may not represent some orthogonal matrices.
- Results reported in the paper use a single GPU type (V100) and limited comparisons; the Double GSOFT theory for convolutions is not proven with the same guarantees.
Experimental setup
Datasets: GLUE DreamBooth CIFAR-100
Benchmarks: GLUE DreamBooth CIFAR-100
Metrics: accuracy Matthews correlation Pearson correlation CLIP-I CLIP-T robust accuracy training time
Baselines: LoRA OFT BOFT full fine-tuning SOC BCOP
Builds on: OFT BOFT Monarch matrices LoRA SOC
Closest papers by meaning
- Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (ICLR 2024 · similarity 0.89)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.87)
- Olica: Efficient Structured Pruning of Large Language Models without Retraining (ICML 2025 · similarity 0.86)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.86)
- Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation (ICML 2024 · similarity 0.85)
- Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.