Research map / Probabilistic and scientific modeling / Variational and generative sampling
Latent Space Factorization in LoRA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FVAE-LoRA, which replaces LoRA's A matrix with a factorized two-latent VAE trained under a modified ELBO with a cross-prior repulsion term, so only the task-salient latent z1 feeds the low-rank update.
Problem
Standard LoRA has no explicit mechanism to ensure the learned low-rank subspace captures task-salient rather than residual or spurious information from pre-training.
Approach
Learn two latent spaces via a VAE attached to each LoRA layer; a novel factorizing ELBO adds a repulsive cross-prior term Gamma that pushes the residual latent z2 away from the task latent z1, and only z1 is passed through B for the adaptation.
Key results
- On six image datasets with ViT-B/16, FVAE-LoRA averages 89.53% vs 87.39% for LoRA and 89.38% for Full FT.
- On commonsense reasoning with Llama-3-8B, FVAE-LoRA averages 87.82% vs 87.40% for HiRA and 77.82% for LoRA.
- On TIMIT, FVAE-LoRA reaches 8.09 PER vs 9.38 for LoRA and 7.48 for Full FT; on Animals worst-group accuracy rises from 54.79 to 62.0 vs LoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LoRA with FVAE-LoRA | ViT-B/16 · Image classification average (DTD, EuroSAT, GTSRB, RESISC45, SUN397, SVHN) | top-1 accuracy: 87.39 -> 89.53 | Helped |
| replace HiRA with FVAE-LoRA | Llama-3-8B · Commonsense reasoning average (PIQA, SIQA, ARC-c, ARC-e, OBQA, HellaSwag, WinoGrande) | accuracy: 87.40 -> 87.82 | Helped |
| replace LoRA with FVAE-LoRA | roberta-base · GLUE subset average | average score: 79.81 -> 81.21 | Helped |
| replace LoRA with FVAE-LoRA | Wav2Vec2-Large · TIMIT | phoneme error rate (PER): 9.38 -> 8.09 | Helped |
| replace LoRA with FVAE-LoRA | ViT-B/16 · Animals | worst-group accuracy (WG): 54.79±8.08 -> 62.0±4.83 | Helped |
| replace LoRA with FVAE-LoRA | ViT-B/16 · Waterbirds | worst-group accuracy (WG): 75.49±0.9 -> 75.85±3.72 | No clear effect |
| replace LoRA with FVAE-LoRA | ViT-B/16 · CelebA | average accuracy: 96.09±0.02 -> 95.77±0.18 | Hurt |
| replace LoRA with FVAE-LoRA | ViT-B/16 · Spurious correlation benchmarks (Animals, Waterbirds, CelebA) | accuracy disparity: 34.8 -> 31.71 | Helped |
| add the FVAE factorization regularizer (ablation) | two-latent VAE LoRA on ViT-B/16 · Image classification average (DTD, EuroSAT, GTSRB, RESISC45, SUN397, SVHN) | top-1 accuracy: 86.43 -> 89.53 | Helped |
| use β-VAE2LAT with β=10 instead of the FVAE factorization objective (ablation) | two-latent VAE LoRA on ViT-B/16 · Image classification average (DTD, EuroSAT, GTSRB, RESISC45, SUN397, SVHN) | top-1 accuracy: 89.53 -> 87.29 | Hurt |
| reduce β to 0.1 (ablation) | FVAE-LoRA · All tasks | task performance | Hurt |
| increase β to 100 (ablation) | FVAE-LoRA · Some tasks | task performance | Helped |
Surprising findings
- FVAE-LoRA's average image accuracy slightly exceeds full fine-tuning (89.53 vs 89.38).
- The authors state the objective is no longer a strict lower bound on log-likelihood, and that naive symmetric KL-repulsion terms performed worse than LoRA.
Limitations
- Adapter cannot be merged into base weights, so dynamic per-input adaptation adds inference cost.
- LoRA rank fixed at 16 and adaptation restricted to query and key matrices.
- Training cost is higher (about 30% over DoRA) due to the VAE decoder.
Experimental setup
Datasets: DTD EuroSAT GTSRB RESISC45 SUN397 SVHN commonsense reasoning training corpus (LLM-Adapters) GLUE subset TIMIT Waterbirds CelebA Animals (SpuCo)
Benchmarks: DTD EuroSAT GTSRB RESISC45 SUN397 SVHN PIQA SIQA ARC-c ARC-e OBQA HellaSwag
Metrics: top-1 accuracy accuracy Matthews correlation Pearson correlation phoneme error rate worst-group accuracy accuracy disparity
Baselines: Full FT LoRA PiSSA rsLoRA DoRA OLoRA HiRA Prompt Tuning P-Tuning ChatGPT
Builds on: LoRA beta-VAE DoRA PiSSA rsLoRA
Closest papers by meaning
- SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.89)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.89)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.88)
- LoRAtorio: An intrinsic approach to LoRA Skill Composition (NeurIPS 2026 · similarity 0.87)
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.