Research map / Vision-language and multimodal / Human and embodied modeling
Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Orthus builds a unified image-text model by replacing VQ with a soft differentiable embedding on top of Chameleon-7B and adding a diffusion head beside the LM head, yielding fully autoregressive interleaved generation with continuous image features.
Problem
Unified multimodal models either lose information by quantizing images into discrete tokens (fully AR models) or suffer from noise disturbance when mixing diffusion with AR (AR-diffusion models), which hampers joint understanding and generation of interleaved image-text content.
Approach
Keep a fully autoregressive transformer over discrete text tokens and continuous image patch features, using a softmax-relaxed (soft VQ) input embedding and a modality-specific diffusion head for next-patch prediction, initialized cheaply from Chameleon-7B by tuning only the embedding, codes and diffusion head.
Key results
- Orthus reaches a GenEval score of 0.58 and MME-P of 1265.8 at 7B, outperforming Chameleon post-trained on the same data (0.43 GenEval, 1056.9 MME-P).
- Orthus-base is built in about 72 A100 GPU hours by tuning only the embedding modules and diffusion head (~0.3B params) on 10k images.
- On visual understanding, Orthus scores POPE 79.6 and GQA 52.8, above Show-o (73.8 and 48.7) and LWM (75.2 and 44.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-train with Orthus’s unified understanding-and-generation objective rather than understanding-only training (ablation) | Orthus · POPE | accuracy: 78.7 -> 79.6 | Helped |
| post-train with Orthus’s unified understanding-and-generation objective rather than understanding-only training (ablation) | Orthus · MME-P | score: 1244.2 -> 1265.8 | Helped |
| post-train with Orthus’s unified understanding-and-generation objective rather than understanding-only training (ablation) | Orthus · GQA | accuracy: 51.9 -> 52.8 | Helped |
| post-train with Orthus’s unified understanding-and-generation objective rather than generation-only training (ablation) | Orthus · GenEval | score: 0.56 -> 0.58 | Helped |
| use softmax differentiable vision embedding instead of argmin VQ embedding (ablation) | Orthus · POPE | accuracy: 77.6 -> 78.7 | Helped |
| use softmax differentiable vision embedding instead of argmin VQ embedding (ablation) | Orthus · MME-P | score: 1064.8 -> 1244.2 | Helped |
| use softmax differentiable vision embedding instead of randomly initialized linear embedding (ablation) | Orthus · POPE | accuracy: 70.4 -> 78.7 | Helped |
| use softmax differentiable vision embedding instead of randomly initialized linear embedding (ablation) | Orthus · MME-P | score: 800.7 -> 1244.2 | Helped |
| use softmax differentiable vision embedding instead of argmin VQ embedding (ablation) | Orthus · VQAv2 | accuracy: 57.9 -> 60.8 | Helped |
| use softmax differentiable vision embedding instead of argmin VQ embedding (ablation) | Orthus · GQA | accuracy: 50.1 -> 51.9 | Helped |
| use softmax differentiable vision embedding instead of argmin VQ embedding (ablation) | Orthus · MMMU | accuracy: 26.7 -> 28.0 | Helped |
| use diffusion loss instead of MSE loss for image-feature prediction (ablation) | Orthus · text-to-image generation | image detail and color diversity | Helped |
Surprising findings
- Replacing the argmin VQ lookup with a randomly initialized linear embedding severely hurts understanding (POPE 70.4 vs 78.7), showing the pretrained code embeddings matter.
Limitations
- Relies on initialization from Chameleon-7B and VQ-VAE codebook, so it inherits that architecture's constraints.
- Understanding results lag larger or specialized models on some benchmarks (e.g., VQAv2 vs NExT-GPT-13B), and scaling is left to future work.
- Diffusion-head sampling (100 DDIM steps with CFG) adds inference cost for image generation.
Experimental setup
Datasets: LAION-Aesthetic LLaVA-v1.5-665K JourneyDB LAION-COCO-aesthetic Instruct-Pix2Pix StoryStream ShareGPT-4V
Benchmarks: POPE MME-P VQAv2 GQA MMMU GenEval HPSv2 Instruct-Pix2Pix CLIP similarity
Metrics: accuracy MME-P score GenEval score HPS CLIP similarity
Baselines: Chameleon Show-o LWM Transfusion SEED-X SDXL Emu3 NExT-GPT LLaVA-v1.5 InstructBLIP
Builds on: Chameleon MAR Transfusion VQ-VAE DDIM
Closest papers by meaning
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.89)
- STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation (NeurIPS 2026 · similarity 0.89)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.88)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.