Research map / Vision-language and multimodal / Multimodal LLMs
HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HermesFlow, which builds homologous understanding and generation preference pairs from the same image/prompt and aligns them jointly with Pair-DPO in self-play iterations, narrowing the understanding-generation gap of Show-o.
Problem
Unified MLLMs show a notable gap where understanding is stronger than generation, and prior data-level or single-capability DPO methods do not jointly close this gap.
Approach
Curate paired understanding and generation preference data from homologous inputs (image-caption pairs), using BERT similarity for captions and self-VQA accuracy (TIFA questions) for images, then optimize both with a Pair-DPO loss that multiplies the two DPO preference margins inside a single sigmoid, iterated in a self-play loop.
Key results
- HermesFlow (1.3B) reaches POPE 81.4, MME 1249.7 and MMMU 28.3 vs Show-o's 80.0, 1232.9 and 27.4 on understanding benchmarks.
- On GenEval overall, HermesFlow scores 0.69 vs Show-o 0.53 and Janus 0.61; DPG-Bench average 70.22 vs 67.48.
- Understanding-generation gap falls from 0.087 (Show-o) to 0.036 (HermesFlow) on the authors' GPT-4o-judged quantitative assessment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply HermesFlow Pair-DPO alignment to Show-o | Show-o · POPE | score: 80.0 -> 81.4 | Helped |
| apply HermesFlow Pair-DPO alignment to Show-o | Show-o · GenEval Overall | score: 0.53 -> 0.69 | Helped |
| apply HermesFlow alignment to Show-o | Show-o · Understanding-generation gap | Gap: 0.087 -> 0.036 | Helped |
| use understanding-only DPO instead of paired preference optimization (ablation) | Show-o · GenEval Overall | score: 0.53 -> 0.58 | Helped |
| use understanding-only DPO instead of paired preference optimization (ablation) | Show-o · POPE | score: 80.0 -> 80.8 | Helped |
| use generation-only DPO instead of paired preference optimization (ablation) | Show-o · POPE | score: 80.0 -> 80.5 | Helped |
| use generation-only DPO instead of paired preference optimization (ablation) | Show-o · GenEval Overall | score: 0.53 -> 0.70 | Helped |
| add paired understanding and generation preference optimization in one round (ablation) | Show-o · GenEval Overall | score: 0.53 -> 0.68 | Helped |
| add a second Pair-DPO self-play iteration (ablation) | Show-o · MMMU | score: 28.0 -> 28.1 | Helped |
| add a third Pair-DPO self-play iteration (ablation) | Show-o · GenEval Overall | score: 0.69 -> 0.69 | No clear effect |
| increase the number of generated preference samples (ablation) | HermesFlow · understanding and generation | performance | Helped |
| use too few generated preference samples (ablation) | HermesFlow · understanding and generation | performance | Hurt |
Surprising findings
- Pair-DPO using only one modality's preference data (understanding or generation) also improves the other modality.
- Generation is more sensitive to preference-data noise than understanding, requiring more samples to match baseline performance.
Limitations
- Only one backbone (Show-o) was tested due to limited open-source MLLMs.
- Generation score relies on GPT-4o judging with strict criteria; the authors note generation capability is limited.
- Pair-DPO objective multiplies the two DPO margins inside one sigmoid, a design choice whose motivation is only lightly justified.
Experimental setup
Datasets: JourneyDB TIFA-generated VQA
Benchmarks: POPE MME Flickr30k VQAv2 GQA MMMU GenEval DPG-Bench MSCOCO-30K FID MSCOCO CLIP-Score
Metrics: accuracy CIDEr overall score average score FID CLIP-Score understanding-generation gap
Baselines: Show-o Janus VILA-U SEED-X Chameleon SD 2.1 LWM DPO (Understanding) DPO (Generation)
Builds on: DPO Show-o TIFA CSR
Closest papers by meaning
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.90)
- On Fairness of Unified Multimodal Large Language Model for Image Generation (NeurIPS 2025 · similarity 0.89)
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.89)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.89)
- STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation (NeurIPS 2026 · similarity 0.89)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.