Research map / Vision-language and multimodal / Multimodal LLMs

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

ICLR 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Kosmos-G uses an MLLM (Kosmos-1) aligned to the CLIP text space via an AlignerNet and instruction-tuned with score-distillation diffusion loss, enabling zero-shot subject-driven generation from interleaved multi-image and text inputs.

Problem

Subject-driven image generation typically requires test-time tuning and cannot accept interleaved multi-image and text inputs, limiting multi-entity composition and integration with existing U-Net pipelines.

Approach

Replace the CLIP text encoder of Stable Diffusion with an MLLM (Kosmos-1 based), align its output space to CLIP via a text-anchored Transformer AlignerNet, then instruction-tune with a compositional generation task using the frozen U-Net diffusion loss as score distillation, so the U-Net needs no modification.

Key results

• On DreamBench, zero-shot Kosmos-G with single image input achieves DINO 0.694, CLIP-I 0.847, CLIP-T 0.287, outperforming Textual Inversion and Re-Imagen and marginally beating DreamBooth and BLIP-Diffusion on DINO/CLIP-I.
• On MS-COCO zero-shot FID, Kosmos-G-1.9B reaches 10.99, better than GILL-8B (12.20) and Emu-14B (11.66) among CLIP-aligned VL2I models but worse than SD v1.5 (9.34) and Imagen (7.27).
• Ablation: end-to-end fine-tuning without AlignerNet fails, while AlignerNet with 24 layers yields FID 9.55 versus 9.34 for SD v1.5.

Measured effects reported in the paper

Surprising findings

• Direct end-to-end fine-tuning of the MLLM through the diffusion U-Net without AlignerNet fails to generate meaningful images.
• Adding a fixed prompt prefix such as 'a' degrades DreamBench editing and customization performance, attributed to BLIP-2 caption style in training data.

Limitations

• Single-image DreamBench results are only comparable to or marginally better than fine-tuning methods; SuTI scores higher on DINO (0.741).
• Text-to-image FID lags behind SD v1.5 and Imagen, and end-to-end training with AlignerNet is costlier and performs worse under equal GPU days.
• Prompt prefixes like 'a' hurt editing and customization, and the instruction data has limited editing examples.

Experimental setup

Datasets: LAION-2B LAION-400M COYO-700M Conceptual Captions Open Images V7 InstructPix2Pix data DreamBench MS-COCO 2014

Benchmarks: DreamBench MS-COCO 2014 validation

Metrics: DINO CLIP-I CLIP-T FID

Baselines: Textual Inversion DreamBooth BLIP-Diffusion Re-Imagen SuTI GLIDE Make-A-Scene DALL-E 2 Imagen GILL Emu

Builds on: Kosmos-1 Stable Diffusion v1.5 Score Distillation Sampling GILL Magneto

Closest papers by meaning

• VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
• Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.89)
• MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.89)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.89)