Research map / Image generation / Autoregressive and latent generation
MaGIC: Multi-modality Guided Image Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaGIC, a training-free consistent modality blending method that combines separately trained single-modality U-Net adapters (MCU-Net) for multi-modality guided image completion.
Problem
Image completion over large masked regions is ambiguous, and existing guided completion methods support only a single guidance modality; joint retraining is needed to combine modalities.
Approach
Train one modality-specific conditional U-Net per guidance type on a frozen Stable Diffusion inpainting denoiser, then fuse arbitrary combinations at inference by gradient guidance in latent space that pulls the frozen U-Net's features toward each MCU-Net's guided features, with converse amplification scale factors.
Key results
- MaGIC with CMB reaches FID 37.65 and PickScore 49.57 on COCO, versus 47.70 / 30.79 for the guidance-free MaGIC and 45.51 FID for MAT.
- Single-modality MCU-Net with depth guidance reaches FID 39.43 on COCO, and segmentation-guided MCU-Net reaches FID 8.32 on Places2, improving over ZITS (18.96).
- CMB outperforms feature-level addition (FLA) on COCO (FID 37.65 vs 37.78 at 35 steps and 41.53 at 50 steps).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add five modalities with CMB instead of no guidance | MaGIC · COCO | FID: 47.70±0.29 -> 37.65±0.22 | Helped |
| add five modalities with CMB instead of no guidance | MaGIC · COCO | PickScore: 30.79±0.10 -> 49.57±0.17 | Helped |
| replace CMB with feature-level addition (ablation) | MaGIC · COCO | PickScore: 49.57±0.17 -> 44.19±0.23 | Hurt |
| increase feature-level addition inference from 35 to 50 steps (ablation) | MaGIC with FLA · COCO | PickScore: 44.19±0.23 -> 35.85±0.08 | Hurt |
| remove text guidance from the five-modality configuration (ablation) | MaGIC with CMB · COCO | FID: 35.42±0.10 -> 37.65±0.22 | Hurt |
| remove text guidance from the five-modality configuration (ablation) | MaGIC with CMB · COCO | PickScore: 47.81±0.34 -> 49.57±0.17 | Hurt |
| use text guidance only instead of five modalities (ablation) | MaGIC with CMB · COCO | FID: 39.96±0.12 -> 37.65±0.22 | Mixed |
| use text guidance only instead of five modalities (ablation) | MaGIC with CMB · COCO | PickScore: 48.58±0.25 -> 49.57±0.17 | Helped |
| increase CMB denoising steps P from 10 to 50 with Q=5 (ablation) | MaGIC with CMB · COCO | FID: 40.40±0.25 -> 36.60±0.12 | Helped |
| increase CMB denoising steps P from 10 to 50 with Q=5 (ablation) | MaGIC with CMB · COCO | PickScore: 45.98±0.12 -> 50.52±0.07 | Helped |
| increase CMB gradient iterations Q from 1 to 5 with P=30 (ablation) | MaGIC with CMB · COCO | FID: 38.50±0.23 -> 37.65±0.22 | Helped |
| increase CMB gradient iterations Q from 5 to 10 with P=30 (ablation) | MaGIC with CMB · COCO | PickScore: 49.57±0.17 -> 48.60±0.12 | Hurt |
Surprising findings
- Naive feature-level addition of multiple modality features degrades quality, with 50 DDIM steps worse than 35 steps.
- A jointly re-trained unified multi-modality model achieves lower FID than the training-free CMB approach, per the authors' joint re-training experiment.
Limitations
- High-frequency detail fidelity is bounded by the backbone completion model.
- Inference is slower than single-step completion models and scales with the number of modalities.
- Inherent Stable Diffusion issues (e.g., misplaced objects, lost depth information) persist when modalities have low correlation.
Experimental setup
Datasets: COCO COCO-Stuff Places2 LAION-Aesthetics
Benchmarks: COCO Places2
Metrics: FID PickScore U-IDS P-IDS CLIP score PSNR SSIM LPIPS
Baselines: EC CTSDG ZITS LAMA LDM MAT ControlNet T2I-Adapter CoMod TFill FcF
Builds on: Stable Diffusion inpainting T2I-Adapter Blended Diffusion DDIM
Closest papers by meaning
- Unleashing Network Potentials for Semantic Scene Completion (CVPR 2024 · similarity 0.85)
- Completion as Enhancement: A Degradation-Aware Selective Image Guided Network for Depth Completion (CVPR 2025 · similarity 0.85)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.85)
- Voxel Proposal Network via Multi-Frame Knowledge Distillation for Semantic Scene Completion (NeurIPS 2024 · similarity 0.84)
- DMF-Net: Image-Guided Point Cloud Completion with Dual-Channel Modality Fusion and Shape-Aware Upsampling Transformer (AAAI 2025 · similarity 0.84)
- Context and Geometry Aware Voxel Transformer for Semantic Scene Completion (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.