Research map / Vision-language and multimodal / Multimodal LLMs
FUSE: Fine-Grained and Semantic-Aware Learning for Unified Image Understanding and Generation
Peng Zhang, Wanggui He, Mushui Liu, Wenyi Xiao, Siyu Zou, Yuan Li, Xingjian Wang, Guanghao Zhang, Yanpeng Liu, Weilong Dai, Jinlong Liu, Shuyi Ying and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FUSE, a unified image understanding and generation model that combines fine-grained and semantic-aware learning to improve fine-grained tasks such as image editing and virtual try-on beyond MLLM-only semantic encoding.
Problem
Unified models that rely heavily on MLLM high-level semantic encoding for diffusion-based image generation struggle with fine-grained visual tasks like image editing and virtual try-on.
Approach
Augments MLLM semantic conditioning with fine-grained visual learning so that the diffusion generator receives both high-level semantic and low-level detail signals; the abstract is truncated so the exact mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| inject VAE features via SD-Connector (ablation) | FUSE · any2any-add subset (zero-shot editing) | editing consistency | Improved |
| remove VAE features at diffusion stage (ablation) | FUSE · any2any-add subset | semantic-level editing | No clear effect |
| use step-wise credit assignment instead of uniform reward weighting (ablation) | Adaptive-GRPO · held-out test set | semantic reward | Improved |
| use step-wise credit assignment instead of uniform reward weighting (ablation) | Adaptive-GRPO · held-out test set | fine-grained reward | Improved |
| use FUSE unified model instead of BAGEL | BAGEL · GenEval | overall score: 0.88 -> 0.89 | Improved |
| use FUSE instead of FLUX.1-dev | FLUX.1-dev · MJHQ-30K | FID: 6.6 -> 5.6 | Improved |
| use FUSE instead of FLUX.1-dev | FLUX.1-dev · WISE | score: 0.50 -> 0.65 | Improved |
| use FUSE instead of UniWorld-V1 | UniWorld-V1 · ImgEdit-Bench | overall: 3.37 -> 3.88 | Improved |
| use FUSE instead of best baselines on position prompts | Janus-Pro-7B · GenEval Position | score: 0.79 -> 0.77 | Hurt |
| use FUSE instead of Step1X-Edit on style edits | Step1X-Edit · ImgEdit-Bench Style | score: 4.44 -> 4.20 | Hurt |
Limitations
- Full text unavailable; only the truncated abstract was available
Closest papers by meaning
- Image Fusion via Vision-Language Model (ICML 2024 · similarity 0.89)
- Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond (CVPR 2025 · similarity 0.89)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
