Research map / Vision-language and multimodal / Multimodal LLMs
Reconstruction Alignment Improves Unified Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RecA, a self-supervised post-training method that conditions a unified multimodal model on its own understanding-encoder embeddings and trains it to reconstruct the input image, improving generation and editing without captions or RL.
Problem
Unified multimodal models trained on sparse image-text captions miss fine-grained visual details, leading to misalignment between understanding and generation (e.g., collapsing 'yellow broccoli' to green).
Approach
Treat the UMM's own visual understanding encoder embeddings as dense 'text prompts' and post-train the generator to reconstruct the input image from them (semantic reconstruction loss), requiring only unlabeled images and no auxiliary modules, distillation data or RL.
Key results
- Harmon-1.5B with RecA reaches GenEval 0.86 and DPGBench 87.21 without distillation data, up from 0.73 and 80.93 baseline.
- BAGEL-RecA improves ImgEdit overall from 3.38 to 3.75 and GEdit-Bench-EN from 6.94 to 7.27 using 10k FLUX images.
- With SFT on BLIP3o-60k followed by RecA on MidjourneyV6, GenEval reaches 90.15 and DPG 88.15.
Surprising findings
- Applying RecA before SFT hurts performance (-3.09 GenEval), so ordering matters: RecA works best as a refinement stage after SFT.
- Higher-resolution understanding embeddings (512) give worse results than low-resolution (224) embeddings, as they leak low-level pixel cues.
Limitations
- Reasoning-intensive generation (WISE) shows limited gains for BAGEL and Show-o.
- Naive high-resolution reconstruction with VQGAN tokens collapses via token copying, requiring input resizing or blurring.
- Reported GenEval gains partly depend on evaluation template leakage in GPT-4o-distilled data, which the authors control for only by ablation.
Experimental setup
Datasets: MidjourneyV6 LLaVA Mix-665K Text-to-Image-2M (FLUX-generated subset) BLIP3o-60k GenEval DPGBench ImgEdit GEdit-Bench-EN WISE
Benchmarks: GenEval DPGBench WISE ImgEdit GEdit-Bench-EN POPE MME GQA MMMU SEED
Metrics: GenEval overall DPG score WISE score ImgEdit overall GEdit overall accuracy
Baselines: Harmon Show-o OpenUni BAGEL Janus-Pro BLIP3-o FLUX-Kontext BAGEL-NHR SFT
Builds on: Show-o Harmon OpenUni BAGEL REPA
Closest papers by meaning
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.89)
- Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.89)
- G$^2$TR: Generation-Guided Visual Token Reduction for Separate-Encoder Unified Multimodal Models (NeurIPS 2026 · similarity 0.88)
- REPA Works Until It Doesn’t: Early-Stopped, Holistic Alignment Supercharges Diffusion Training (NeurIPS 2025 · similarity 0.88)
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models (NeurIPS 2025 · similarity 0.88)
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.