Research map / Vision-language and multimodal / Multimodal LLMs

SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SynerGen-VL, an encoder-free unified MLLM that uses token folding over discrete VQ image tokens, image-specific vision-expert FFNs, and two-stage progressive alignment pretraining to do image understanding and generation with one next-token objective.

Problem

Unified MLLMs for image understanding and generation either rely on complex designs (external diffusion models, separate encoders, semantic tokenizer pretraining) or, when encoder-free, suffer from training instability/cost, interference with LLM knowledge, and long visual token sequences for high-r

Approach

Fold m×n groups of discrete VQ tokens into single visual tokens before the LLM to shorten sequences (2×8 folding), add image-specific vision-expert FFNs initialized from text FFNs with the LLM frozen in stage 1, and unfold outputs via a shallow autoregressive head for generation.

Key results

• On GenEval, SynerGen-VL (2.4B active) reaches 0.61 overall, tied with Janus (1.3B) and above Emu3-Gen (8B, 0.54).
• On MSCOCO-30K zero-shot FID, SynerGen-VL achieves 7.65 versus Emu3-Gen's 19.3 with fewer parameters; on MJHQ-30K it reaches 6.10.
• Token folding raises TextVQA from 18.7 to 35.0 and DocVQA from 14.7 to 36.7 at the same image token length (1024) in the ablation.

Measured effects reported in the paper

Surprising findings

• Progressive alignment with only stage 1 yields near-zero VQA scores (e.g., DocVQA 0.2), showing stage 1 alone does not give usable understanding.
• Full-parameter tuning degrades NLP benchmarks sharply (MMLU 42.3 baseline to 23.1), while progressive tuning retains 39.3.

Limitations

• Understanding performance trails specialized encoder-based models on some benchmarks such as MMBench (53.7) and MMMU is competitive but MathVista/AI2D lag behind (AI2D 60.8).
• Stage 1 alone does not yield usable understanding, so the two-stage schedule is required.
• Compute hardware and training duration beyond step counts are not stated in the main text; code and models are only promised ('shall be released').

Experimental setup

Datasets: LAION-Aesthetics Megalith SAM Objects365 ImageNet-1k LAION-En COYO Laion-Zh GRIT COCO TextCaps All-Seeing

Benchmarks: MMBench MMVet MMMU MME MathVista POPE SEED-Image OCRBench TextVQA ScienceQA GQA DocVQA

Metrics: accuracy FID GenEval overall score

Baselines: Emu3-Chat Emu3-Gen Janus Show-o VILA-U LWM SEED-X Chameleon LLaVA-1.5 EVE-7B Mono-InternVL SDXL

Builds on: Emu3 InternLM2 InternVL-1.5 Multimodal Mixture-of-Experts Pixel Shuffle

Closest papers by meaning

• Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.90)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.90)
• UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation (NeurIPS 2025 · similarity 0.90)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.89)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)