Research map / Architectures and efficiency / Pruning and efficient tuning

LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LOFT, a low-rank orthogonal PEFT framework that separates the adaptation support from the in-subspace transform, and selects supports from downstream gradients (GradSVD, SkewGrad) rather than pretrained principal directions.

Problem

Orthogonal PEFT methods couple the adaptation subspace with the transform parameterization, so the effect of the support choice is confounded and rarely studied; existing supports are chosen task-agnostically from pretrained structure.

Approach

Write orthogonal adaptation as a multiplicative subspace rotation W(I + P_r^T(T_r - I)P_r) with separate support P_r and transform T_r; show a first-order result that the initial gradient signal is governed by skew(W_0^T G), and choose the support from downstream gradients (SkewGrad top invariant subspace, GradSVD top right singular subspace of G).

Key results

• On GLUE (DeBERTaV3-base, six-task average over five seeds), LOFT with SkewGrad support reaches 88.73 vs. 88.04 for PSOFT with 0.07M parameters.
• On VTAB-1K (ViT-B/16), LOFT with SkewGrad support reaches 73.8 average vs. 73.4 for PSOFT, with 5.7GB peak memory vs. 6.2GB.
• On GSM8K with a ~0.5M parameter budget, LOFT with GradSVD support reaches 40.94 vs. 35.94 for PSOFT, while all LOFT variants use 25.68GB peak memory.

Surprising findings

• Principal-support LOFT at rank 46 does not improve the VTAB average over PSOFT, suggesting gains come from support selection rather than rank.
• Free (unconstrained) in-subspace transforms help on some sentence-pair tasks but are not uniformly superior to orthogonal transforms.

Limitations

• Focuses only on support choice; uses the default Cayley transform and does not explore other transform parameterizations.
• Preferred gradient-informed support varies by task family (SkewGrad in encoder/vision, GradSVD more stable in some decoder and multilingual settings).
• Gradient calibration adds a one-off step that slightly increases end-to-end runtime versus principal support.

Experimental setup

Datasets: GLUE (CoLA, STS-B, RTE, MRPC, SST-2, QNLI) VTAB-1K MetaMathQA-40K Bactrian-X MNLI QQP

Benchmarks: GLUE VTAB-1K GSM8K MATH MNLI QQP

Metrics: Matthews correlation Pearson correlation accuracy peak GPU memory trainable parameters

Baselines: FFT GOFTv2 qGOFTv2 BOFT OFTv2 LoRA PiSSA DoRA LoRA-XS PSOFT HRA

Builds on: OFT PSOFT LoRA BOFT HRA

Closest papers by meaning

• Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation (ICML 2024 · similarity 0.89)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation (NeurIPS 2024 · similarity 0.87)
• Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization (ICLR 2024 · similarity 0.87)
• PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
• From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.87)