Research map / Image generation / Autoregressive and latent generation
HyperLoRA: Parameter-Efficient Adaptive Generation for Portrait Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperLoRA, a zero-shot portrait personalization method that uses a perceiver-resampler hypernetwork to generate LoRA weights (via a linear LoRA basis) from face images, split into ID and Base LoRA parts, built on SDXL.
Problem
Personalized portrait synthesis needs high fidelity, editability and fast inference; tuning-based LoRA/DreamBooth require per-identity training, while adapter methods like IP-Adapter produce unnatural, less photorealistic faces.
Approach
Instead of injecting identity tokens through extra cross-attention as adapters do, a hypernetwork directly generates LoRA weights as coefficients over a low-dimensional linear LoRA basis, decomposed into an ID-LoRA and a Base-LoRA trained in a multi-stage schedule.
Key results
- Ours (Full) achieves CLIP-I 0.853 for fidelity, above IP-Adapter 0.764, InstantID 0.734, PuLID 0.771 and Arc2Face 0.786.
- Ours (ID) reaches CLIP-T 0.252 and CLIP-I 0.748 for editability, second only to PuLID (0.805 CLIP-I, 0.259 CLIP-T) in editability.
- Ours (Full) gets ID similarity 0.678, the highest among methods except InstantID (0.681).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HyperLoRA with ID-LoRA and Hyper Base-LoRA | LEOSAM portrait synthesis · Unsplash-50 | CLIP-I (fidelity): 0.771 -> 0.853 | Helped |
| Add Hyper Base-LoRA to ID-LoRA (ablation) | HyperLoRA · Unsplash-50 | CLIP-I (fidelity): 0.831 -> 0.853 | Helped |
| Add Hyper Base-LoRA to ID-LoRA (ablation) | HyperLoRA · Unsplash-50 | CLIP-I (editability): 0.748 -> 0.710 | Hurt |
| Add Hyper Base-LoRA during training (ablation) | HyperLoRA ID-LoRA · portrait prompt adherence | prompt adherence | Helped |
| Train with ID embeddings in addition to CLIP features (ablation) | HyperLoRA ID-LoRA · portrait editing with glasses and mask prompts | prompt adherence | Helped |
| Average predicted LoRA coefficients from multiple input images instead of using one input (ablation) | HyperLoRA · multiple-input portrait synthesis | ID Sim.: 0.65 -> 0.72 | Helped |
Surprising findings
- Adapter-based methods require lower CFG scales while HyperLoRA works across CFG 3 to 7, which the authors attribute to LoRA-style injection.
Limitations
- Limited to LoRA rank 8 for ID-LoRA and rank 4 for Base-LoRA due to GPU memory constraints.
- Base-LoRA leaks identity-irrelevant information and trades off editability against fidelity; editability is lower than PuLID.
- Training requires about 10 days on 16 A100 GPUs.
Experimental setup
Datasets: LAION-2B subset (4.4M portraits) Unsplash-50
Benchmarks: Unsplash-50
Metrics: CLIP-I ID similarity CLIP-T
Baselines: IP-Adapter InstantID PuLID Arc2Face
Builds on: LoRA IP-Adapter SDXL DDPM
Closest papers by meaning
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.85)
- EchoShot: Multi-Shot Portrait Video Generation (NeurIPS 2025 · similarity 0.85)
- MagiCapture: High-Resolution Multi-Concept Portrait Customization (AAAI 2024 · similarity 0.85)
- VisionDirector: Vision-Language Guided Closed-Loop Refinement for Generative Image Synthesis (CVPR 2026 · similarity 0.85)
- CapHuman: Capture Your Moments in Parallel Universes (CVPR 2024 · similarity 0.84)
- LoRAShop: Training-Free Multi-Concept Image Generation and Editing with Rectified Flow Transformers (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.