Research map / Vision-language and multimodal / Multimodal LLMs
FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FUDOKI, a unified multimodal model for visual understanding and text-to-image generation trained entirely with discrete flow matching using metric-induced probability paths and kinetic-optimal velocities, initialized from Janus-1.5B.
Problem
Most unified multimodal LLMs use autoregressive architectures with limits in complex reasoning, planning, and self-correction; masked discrete diffusion alternatives cannot revise tokens once unmasked. The paper asks whether discrete flow matching can serve as a unified backbone for understanding an
Approach
Replaces masking-based corruption with a metric-induced probability path (softmax over embedding distances with a schedule beta_t) and kinetic-optimal velocities, so tokens can be revised toward semantically similar states during iterative sampling; the model is initialized from an AR MLLM (Janus-1.5B) and adapted with a two-stage discrete flow matching training.
Key results
- FUDOKI reaches 0.77 overall on GenEval, rising to 0.88 with inference scaling (top 4 of 32 candidates selected by VILA-Judge).
- FUDOKI (1.5B) scores 86.1 POPE, 1485.4 MME-P, 73.9 MMB, 68.2 SEED, 57.6 GQA, 34.3 MMMU, 38.0 MM-Vet; MM-Vet reaches 55.5 with inference scaling (best of 8).
- On DPG-Bench, FUDOKI achieves 83.63 overall, above Janus-Pro-1B (82.63) and the other compared models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply test-time inference scaling by selecting among candidate outputs (ablation) | FUDOKI · GenEval | Overall: 0.77 -> 0.88 | Helped |
| Apply test-time inference scaling by selecting among candidate responses (ablation) | FUDOKI · MM-Vet | score: 38.0 -> 55.5 | Helped |
| Initialize from pretrained AR weights rather than train from scratch (ablation) | FUDOKI · Training | training loss | Helped |
| Remove time-embedding layers (ablation) | FUDOKI · Training | training loss | Helped |
| Increase the number of inference timesteps (ablation) | FUDOKI · Visual generation | speed | Hurt |
| Increase the number of inference timesteps (ablation) | FUDOKI · Visual generation | GenEval score | Helped |
Surprising findings
- Removing explicit time-embedding layers gives slightly lower training loss, suggesting the model implicitly infers the timestep from the corrupted input.
Limitations
- Fixed sequence length required before sampling.
- Weak on complex prompts and text rendering in generated images.
- Limited expert-level reasoning and domain knowledge in understanding.
Experimental setup
Datasets: in-house text-to-image generation data (8.76M) ShareGPT-4o ALLaVA-Instruct ShareGPT4V DocVQA ChartQA DVQA MAVIS GeoQA+ MathInstruct Evol-Instruct
Benchmarks: GenEval DPG-Bench POPE MME-P MMB SEED GQA MMMU MM-Vet
Metrics: GenEval overall score DPG-Bench overall POPE MME-P MMB SEED GQA MMMU MM-Vet
Baselines: Janus-Pro-1B Janus D-DiT UniDisc Show-o Transfusion Emu3-Gen SD3-Medium
Builds on: Discrete Flow Matching Discrete Flow Models Janus LlamaGen SigLIP MaskGIT
Closest papers by meaning
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.89)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.88)
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.88)
- Show-o2: Improved Native Unified Multimodal Models (NeurIPS 2025 · similarity 0.88)
- MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.88)
- Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.