Research map / Architectures and efficiency / Pruning and efficient tuning

ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ETHER and ETHER+, parameter-efficient finetuning methods that use Householder hyperplane reflections (and a relaxed two-sided variant) to bound weight-space distance from the pretrained model, yielding strong accuracy with 10-100x fewer parameters than LoRA/OFT and high learning-rate robustness.

Problem

Parameter-efficient finetuning of large foundation models must balance adaptation against retaining pretrained capabilities, but existing methods (LoRA, OFT) add many parameters and need careful, low learning-rate hyperparameter tuning to avoid deterioration.

Approach

Replace multiplicative orthogonal finetuning matrices with Householder hyperplane reflections (H = I - 2uu^T), which keep distance to identity constant and need only one unit vector per block; the relaxed ETHER+ uses I - uu^T + vv^T applied on both sides of the weight matrix for finer control with bounded distance.

Key results

• On GLUE with DeBERTaV3-base, ETHER+ reaches 90.10 average with 0.33M parameters, versus 89.89 for BOFT (0.75M) and 89.77 for OFT.
• On S2I controllable generation, ETHER uses 0.1M parameters versus 13.2M for OFT and matches its mIoU (24.6 vs 24.5), while ETHER+ reaches 27.3 mIoU and 68.1 Acc.
• On Llama-2-7B instruction tuning, ETHER+ (1.04M params) reaches 44.87 MMLU and 46.50 ARC, beating LoRA rank 8 (4.19M params, 43.61 MMLU).

Measured effects reported in the paper

Surprising findings

• Naive non-orthogonal OFT performs about the same as OFT, suggesting hyperspherical energy retention is not the key factor for finetuning performance.
• ETHER+ alters hyperspherical energy more than Naive, yet achieves the best performance, questioning the role of HE retention.

Limitations

• Expressive power is harder to scale than LoRA's rank parameter, so near full-finetuning performance is not easily approached.
• Multiplicative methods incur higher training computational overhead than additive methods like LoRA, still lagging despite block-parallelism.
• ETHER performs worse on subject-driven generation image quality (DINO 0.567) than other methods, requiring the relaxed ETHER+.

Experimental setup

Datasets: DreamBooth subjects ADE20K GLUE Alpaca CelebA-HQ COCO 2017 VTAB

Benchmarks: DreamBooth subject-driven generation ADE20K semantic map to image (S2I) GLUE MMLU ARC TruthfulQA VTAB

Metrics: DINO CLIP-I CLIP-T LPIPS mIoU Accuracy FID accuracy Matthews correlation MMLU

Baselines: LoRA OFT DreamBooth BOFT AdaLoRA VeRA BitFit Adapters (H-Adapter, P-Adapter) Naive

Builds on: OFT LoRA BOFT VeRA DreamBooth

Closest papers by meaning

• Decoupling Angles and Strength in Low-rank Adaptation (ICLR 2025 · similarity 0.87)
• GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.87)
• Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.87)
• Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models (NeurIPS 2024 · similarity 0.87)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.86)
• PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.86)