Research map / Vision-language and multimodal / Multimodal LLMs
ORION: Decoupling and Alignment for Unified Autoregressive Understanding and Generation
Taihang Hu, Mengting Chen, Jinsong Lan, Xiaoyong Zhu, Kaifu Zhang, Ming-Ming Cheng, Bo Zheng, Yaxing Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ORION, a unified autoregressive multimodal LLM that decouples and aligns understanding and generation to resolve the semantic-structural conflict in monolithic architectures.
Problem
Monolithic autoregressive unified multimodal LLMs suffer a semantic-structural conflict: optimizing for low-level reconstructability in generation causes catastrophic forgetting of high-level semantic understanding.
Approach
Decouples the understanding and generation pathways within a unified autoregressive MLLM and aligns them, rather than using a single monolithic objective, so generation's reconstruction pressure does not erode semantic understanding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add representation consistency loss L_KL (ablation) | ORION (MLP head, 20M warm-up) · MMBench | accuracy: 79.8 -> 83.7 | Improved |
| add representation consistency loss L_KL (ablation) | ORION (MLP head, 20M warm-up) · MMStar | accuracy: 61.0 -> 63.2 | Improved |
| add representation consistency loss L_KL (ablation) | ORION (MLP head, 20M warm-up) · GenEval | overall score: 0.79 -> 0.82 | Improved |
| replace linear vision head with MLP head (ablation) | ORION without L_KL (20M warm-up) · MMBench | accuracy: 71.5 -> 79.8 | Improved |
| replace linear vision head with MLP head (ablation) | ORION without L_KL (20M warm-up) · GenEval | overall score: 0.65 -> 0.79 | Improved |
| MLP head instead of Q-Former at 20M warm-up data (ablation) | ORION without L_KL · MMBench | accuracy: 78.6 -> 79.8 | Improved |
| MLP head instead of Q-Former at 5M warm-up data (ablation) | ORION without L_KL · MMBench | accuracy: 77.3 -> 76.4 | Hurt |
| scale vision-head warm-up data from 5M to 20M (ablation) | ORION MLP head without L_KL · GenEval | overall score: 0.76 -> 0.79 | Improved |
| skip vision-head warm-up (no stage-1 data) (ablation) | ORION MLP head without L_KL · MMBench | accuracy: 76.4 (5M) vs 74.1 (none) | Hurt |
| generative fine-tune of Qwen2.5-VL vs base understanding | ORION · MMBench | accuracy: 79.1 -> 83.7 (MMBench), 67.1 -> 64.5 (MM-Vet) | Mixed |
Closest papers by meaning
- HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation (NeurIPS 2025 · similarity 0.89)
- Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation (CVPR 2026 · similarity 0.89)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.88)
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.87)
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
