Research map / Architectures and efficiency / Pruning and efficient tuning
WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WeGeFT, which fine-tunes a frozen Transformer by generating weight residuals from the pretrained weights via two shared low-rank linear maps, matching or beating LoRA and its variants with fewer parameters and LoRA-level compute and memory.
Problem
Existing PEFT methods trade off parameter, compute, memory efficiency and performance: LoRA variants such as DoRA and VeRA lose compute/memory efficiency, while ReFT-style representation editing and shared-parameter methods like Tied LoRA and VB-LoRA underperform.
Approach
Generate fine-tuning weight residuals directly from the frozen pretrained weights: W_hat = W (I + phi psi), with phi and psi low-rank and shared across layers, so the update is multiplicative and weight-aware; it can be applied equivalently to input activations, keeping LoRA-level cost and needing no token-intervention search.
Key results
- On GSM8K after MetaMathQA fine-tuning of Llama 2 7B, WeGeFT d_in reaches 54.89% with 0.068% trainable parameters, above full fine-tuning (54.20) and DoRA (53.07).
- On Commonsense170k with Llama 3 8B, WeGeFT d_in reaches 87.7 average accuracy with 0.049% parameters, above DoRA (87.6) and LoRA (86.3).
- On FGVC with ViT-B/16, WeGeFT d_in averages 89.39 with 0.025M parameters versus LoRA's 88.71 with 0.147M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WeGeFT d_in instead of LoRA | Llama 2 7B fine-tuned on MetaMathQA · GSM8K test | accuracy: 42.08 -> 54.89 | Helped |
| use WeGeFT d_in instead of LoRA r=16 | LLaMA-1 7B fine-tuned on Math10k · Math10k | average accuracy: 50.9 -> 50.0 | Hurt |
| use WeGeFT d_in instead of DoRA | Llama 3 8B fine-tuned on Commonsense170k · Commonsense170k | average accuracy: 87.6 -> 87.7 | Helped |
| use WeGeFT d_in instead of DoRA | LLaMA-1 7B and Llama 2 7B fine-tuned on Commonsense170k · Commonsense170k | average accuracy: 83.1 -> 82.6; 84.4 -> 83.8 | Hurt |
| use WeGeFT d_in instead of LoRA | ViT-B/16 fine-tuned on FGVC · FGVC | average Top-1 accuracy: 88.71 -> 89.39 | Helped |
| use WeGeFT d_in instead of LoRA | ViT-B/16 fine-tuned on VTAB · VTAB | average accuracy: 74.0 -> 74.1 | Helped |
| replace identity g with nonlinear transformations (ablation) | WeGeFT on ViT-B/16 · FGVC | average Top-1 accuracy: 89.39 -> 88.82 (Sigmoid); 88.94 (GeLU); 88.84 (MLP); 89.12 (Transformer); 89.25 (MLP Mixer) | Hurt |
| replace weight-aware WeGeFT with layer-shared LoRA factors (ablation) | LLaMA-1 7B, Llama 2 7B, and Llama 3 8B · Commonsense170k | average accuracy: 82.6 -> 78.0; 83.8 -> 78.3; 87.7 -> 76.1 | Hurt |
| reduce VeRA intermediate rank from 12288 to 1024 (ablation) | VeRA on Math10k · Math10k | average accuracy: 47.2 -> 45.2; 50.4 -> 48.6 | Hurt |
| increase WeGeFT rank from 64 to 128 under float16 (ablation) | LLaMA-1 7B fine-tuned on Math10k · Math10k | average accuracy: 49.3 -> 50.5 | Helped |
Surprising findings
- A non-linear transformation g_theta in the low-rank latent space hurt performance on FGVC, so the identity (purely linear) form was best.
- On Commonsense170k with LLaMA-1 and Llama 2, DoRA (about 8x more parameters) outperforms WeGeFT, so the advantage depends on the pretrained model's strength.
Limitations
- On LLaMA-1 and Llama 2 commonsense reasoning, WeGeFT trails DoRA, which uses about 8x more parameters.
- Non-linear variants of the latent transformation were only studied on small models and simpler tasks due to compute constraints.
- The authors give no theoretical account of why the linear identity form works better than non-linear alternatives.
Experimental setup
Datasets: Math10k MetaMathQA Commonsense170k WizardLM Code-Feedback VTAB-1k CUB-200 NABirds Oxford Flowers Stanford Cars Stanford Dogs ImageNet21k
Benchmarks: GSM8K AQuA MAWPS SVAMP BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Metrics: accuracy Top-1 accuracy pass@1 GPU memory wall time trainable parameter percentage
Baselines: LoRA DoRA VeRA VB-LoRA FacT DiReFT LoReFT PiSSA rsLoRA LoRA+ AdaLoRA LoRA-GA
Builds on: LoRA ReFT DiReFT Tied LoRA VB-LoRA
Closest papers by meaning
- Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models (ICLR 2025 · similarity 0.89)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.88)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.88)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.87)
- Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models (NeurIPS 2026 · similarity 0.87)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.