Research map / Vision-language and multimodal / Human and embodied modeling

MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis

AAAI 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MARS, an auto-regressive text-to-image model built on frozen Qwen-7B text experts plus trainable visual experts (SemVIE), trained in three stages to reach strong MS-COCO FID at a fraction of SD1.5's GPU cost.

Problem

Auto-regressive text-to-image models suffer from distribution shift when adapting LLMs to image-text data, risking catastrophic forgetting of language abilities and weaker image quality than diffusion models.

Approach

SemVIE adds parallel trainable visual experts to each attention and FFN block of a frozen pre-trained LLM, routing text and visual tokens to separate experts, with a vision head alongside the text head, plus a three-stage training curriculum from 256px to 1024px using fine-grained CogVLM recaptions.

Key results

• MARS reaches FID-30K 6.92 and CLIPScore 32.33 on zero-shot MS-COCO at 256px, versus 9.22 FID for SDv1.5, using 587 vs 6250 A100 GPU days.
• On T2I-CompBench, MARS scores 69.13 color and 71.23 texture, slightly above PixArt-alpha (68.86 and 70.44).
• Removing the visual expert raises FID from 8.24 to 10.13 in the Stage-I ablation.

Measured effects reported in the paper

Surprising findings

• The FID-30K 3.51 result for MARS* is obtained by selecting the best-CLIP-score images from generations, which the paper notes is a cherry-picked setting.

Limitations

• Headline FID 3.51 uses best-CLIP-score selection, not a standard protocol.
• Training data (about 200M pairs, including 50M in-house) is not released, limiting reproducibility.
• Human study is small (30 to 60 participants, 100 prompts) and details are sparse.

Experimental setup

Datasets: LAION-400M CC3M CC12M LAION-COCO COYO Datacomp in-house Chinese image-text data MS-COCO 2014 (evaluation) T2I-CompBench

Benchmarks: MS-COCO 2014 (30K FID, CLIPScore) T2I-CompBench User study

Metrics: FID-30K CLIPScore attribute binding accuracy user preference

Baselines: SDv1.5 Parti DALL-E 2 PixArt-alpha SDXL LAVIT UIO-2 GILL Emu CM3Leon MUSE Imagen

Builds on: Qwen-7B VQ-GAN CogVLM Parti Next token prediction

Closest papers by meaning

• Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.90)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
• Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
• Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.89)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.89)