Research map / Label-efficient and robust learning / Recognition and long-tail
Multi-concept Model Immunization through Differentiable Model Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIMA, a bi-level differentiable model-merging method that meta-learns one immunized Stable Diffusion initialization resistant to LoRA, DreamBooth, Textual Inversion and Custom Diffusion re-learning/personalization on multiple concepts at once.
Problem
Model immunization against malicious adaptation of open-sourced text-to-image models has only been studied for a single harmful concept, while real deployments need protection against multiple concepts simultaneously.
Approach
Treat the merging of per-concept fine-tuned weights (a constrained least-squares merge of cross-attention key/value projections, plus averaging of other weights) as a differentiable optimization layer, and backpropagate through it in a bi-level objective that maximizes the diffusion loss of adaptation on every target concept.
Key results
- On artistic-style re-learning with LoRA, MIMA improves MSGR by 18.95% over Joint training and 10.94% over Compose averaged across groups and metrics.
- On personalization, MIMA improves MSGR by 3.75% over Joint and 6.36% over Compose; on MRSGR it improves by 5.89% and 9.31%.
- On object re-learning, MIMA reduces average target classification accuracy to 1.42% versus 39.0% without immunization, with other-object accuracy 77.1% vs 83.9% for Joint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MIMA instead of JT for multi-concept style re-learning immunization | Stable Diffusion V1-4 · Artistic styles re-learning with UCE and LoRA | MSGR: 18.95% | Helped |
| Use MIMA instead of CP for multi-concept object re-learning immunization | Stable Diffusion V1-4 · Object re-learning with UCE and LoRA | MSGR: 14.82% | Helped |
| Use MIMA instead of JT for multi-concept personalization immunization | Stable Diffusion V1-4 · Personalization across four adaptation methods | MSGR: 3.75% | Helped |
| Use MIMA instead of CP for preserving other personalized concepts | Stable Diffusion V1-4 · Personalization across four adaptation methods | MRSGR: 9.31% | Helped |
| Use MIMA instead of no immunization | Stable Diffusion V1-4 · Object re-learning; target concepts | accuracy: 39.0 -> 1.42 | Helped |
| Use MIMA instead of JT to preserve other-object generation | Stable Diffusion V1-4 · Object re-learning; other concepts | accuracy: 83.9 -> 77.1 | Hurt |
Surprising findings
- Sequentially running IMMA on multiple concepts degrades generation of other concepts as well as the targets, so it is not a viable multi-concept strategy.
- Compose and Joint baselines show negative MSGR in several settings, indicating immunization can fail or even help adaptation for some concept sets.
Limitations
- Evaluation limited to Stable Diffusion V1-4 and UCE-erased models with a few adaptation methods; results are from a single seed run.
- Bi-level optimization with unrolled gradients on a single-step inner loop is an approximation; cost scales with the number of concepts.
- Several baseline settings show negative or inconsistent gaps, and per-group results vary widely across concept sets.
Experimental setup
Datasets: ImageNet subset (10 classes) eight artistic styles (as in UCE) CustomConcept101
Benchmarks: artistic style re-learning (UCE erased styles) object re-learning (ImageNet subset) CustomConcept101 personalization
Metrics: MSGR (mean similarity gap ratio) with CLIP, DINO, LPIPS MRSGR (relative similarity gap ratio on other concepts) ResNet50 classification accuracy
Baselines: Joint (JT) Compose (CP) Erased (UCE without immunization)
Builds on: IMMA Custom Diffusion UCE DreamBooth
Closest papers by meaning
- Model Immunization from a Condition Number Perspective (ICML 2025 · similarity 0.87)
- Parameter Competition Balancing for Model Merging (NeurIPS 2024 · similarity 0.81)
- AdaMerging: Adaptive Model Merging for Multi-Task Learning (ICLR 2024 · similarity 0.81)
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs (NeurIPS 2025 · similarity 0.81)
- Orthogonal Adaptation for Modular Customization of Diffusion Models (CVPR 2024 · similarity 0.81)
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.