Research map / Image generation / Text-to-image diffusion
Recovering the Pre-Fine-Tuning Weights of Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the pre-fine-tuning weights of LoRA-fine-tuned generative and vision models can be recovered from several public LoRA variants via iterative low-rank factorization (Spectral DeTuning) with a rank scheduler.
Problem
Pre-fine-tuning weights of models are assumed unrecoverable after alignment/personalization fine-tuning; the paper asks whether multiple LoRA fine-tuned variants of the same source model leak the original weights.
Approach
Model the fine-tuned weights as a shared matrix plus per-model low-rank residuals and solve alternating truncated-SVD (M-step) and averaging (W-step) updates, with a rank scheduler that increases the allowed residual rank on plateau; ranks are estimated via pairwise difference ranks and an LP.
Key results
- On ViT, Spectral DeTuning reaches W-Error -15.942 versus -5.214 for Mean LoRA and activation distance 1e-6 versus 5e-2.
- On Stable Diffusion 1.5 with n=5 civitai LoRAs, LPIPS drops to 0.009 versus 0.482 for Mean LoRA.
- On Mistral-7B DPO with n=8 LoRAs, W-Error reaches -22.062 versus -10.757 for Mean LoRA, and recovers the Pre-FT model in under five minutes on RTX2080 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Spectral DeTuning instead of Mean LoRA | ViT weight recovery · ViT | W-Error: -5.214 -> -15.942 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | ViT weight recovery · ViT | Act.-Dist.: 5e{-}2 -> 1e{-}6 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Stable Diffusion 1.5 personalization recovery · Stable Diffusion 1.5 personalization | W-Error: -7.540 -> -17.816 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Stable Diffusion 1.5 personalization recovery · Stable Diffusion 1.5 personalization | LPIPS: 0.482 -> 0.009 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Mistral-7B SFT recovery · Mistral-7B SFT | W-Error: -9.299 -> -16.502 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Mistral-7B SFT recovery · Mistral-7B SFT | SBERT: -1.007 -> -9.324 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Mistral-7B DPO recovery · Mistral-7B DPO | W-Error: -10.757 -> -22.062 | Helped |
| Use Spectral DeTuning instead of Mean LoRA | Mistral-7B DPO recovery · Mistral-7B DPO | SBERT: -3.455 -> -14.708 | Helped |
| Add the rank scheduler (ablation) | Spectral DeTuning on Stable Diffusion · Stable Diffusion 1.5 personalization | W-Error convergence | Helped |
| Use varying LoRA ranks with a dedicated rank scheduler for each LoRA (ablation) | Spectral DeTuning · Stable Diffusion 1.5, varying ranks [8,32,32,32,64,100] | LPIPS: 0.009 -> 0.073 | Hurt |
| Estimate LoRA ranks using the rank-estimation heuristic | Spectral DeTuning · Hundreds of LoRA combinations with different ranks | rank estimation accuracy: 100% | Helped |
Surprising findings
- Pre-fine-tuning weights of an aligned Mistral-7B can be recovered from only public LoRA variants without any data or inference.
- Simple averaging of LoRA-fine-tuned weights performs only marginally better than a single LoRA model.
Limitations
- Requires several (5-12) LoRA variants sharing the same source model, and recovery quality depends on the number of LoRAs.
- Only evaluated on LoRA fine-tuning; full fine-tuning and other PEFT methods are not covered.
- No known defense; the paper proposes no mitigation.
Experimental setup
Datasets: VTAB-1K ImageNet-1K UltraChat UltraFeedback COCO Captions AlpacaFarm evaluation
Benchmarks: LoWRA Bench
Metrics: W-Error (log10 MSE) Activation cosine distance LPIPS SBERT log cosine distance
Baselines: LoRA FT Mean LoRA
Builds on: LoRA VTAB-1K DPO
Closest papers by meaning
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.86)
- LoRA vs Full Fine-tuning: An Illusion of Equivalence (NeurIPS 2025 · similarity 0.85)
- Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models (CVPR 2025 · similarity 0.85)
- Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning (NeurIPS 2025 · similarity 0.85)
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.85)
- Spectral Adapter: Fine-Tuning in Spectral Space (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.