Research map / Vision-language and multimodal / Multimodal LLMs

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SILMM, a model-agnostic iterative self-improvement framework where LMMs self-generate compositional prompts, self-assess text-image alignment via decomposed VQA, and are tuned with DPO (discrete) or a kernel-based continuous DPO (KC-DPO, with DropDiv for diversity) to improve compositional text-to-image generation.

Problem

Text-image alignment for compositional prompts (attributes, counting, spatial and semantic relations) remains poor in large multimodal models, and existing fixes rely on prompt engineering, human feedback, or external reward models. DPO is hard to apply to LMMs with continuous visual features becaus

Approach

LMMs generate their own compositional prompts, sample diverse visual representations (nucleus sampling for discrete tokens; dropout-based DropDiv for continuous features), score alignment via decomposed yes/no VQA probability differences, and optimize with DPO. For continuous features, the likelihood is approximated by an isotropic Gaussian so that the DPO objective reduces to a kernel-based dista

Key results

• SILMM on DreamLLM raises DPG-Bench overall from 53.93 to 64.22 (about 19%) and SEED-LLaMA from 47.12 to 57.31 (about 22%).
• On T2I-CompBench++ attribute, SEED-LLaMA improves from 19.20 to 39.60 (about 106%) and DreamLLM from 22.94 to 39.94 (about 74%).
• Emu3 with SILMM improves DPG-Bench overall from 74.19 to 77.45 and TIFA from 81.86 to 85.11.

Measured effects reported in the paper

Surprising findings

• Continuous-feature LMMs (DreamLLM) benefited less from SILMM than discrete ones (SEED-LLaMA), which the authors attribute to the larger, denser continuous feature space needing more data pairs.
• Ablation shows that MaxPool + cosine kernel gives large gains on attribute categories but hurts non-spatial and complex categories (18.72 and 12.27 versus baseline 28.76 and 23.01).

Limitations

• Improvements are smaller on layout, relation and complex categories, where base generation ability is weak and high-quality chosen samples are hard to obtain.
• Higher dropout rates increase diversity but can degrade alignment quality, so the diversity-quality trade-off is hard to balance.
• Self-feedback quality depends on the LMM's own compositional reasoning, which the authors note is still weak.

Experimental setup

Datasets: self-generated compositional prompts (16k per iteration)

Benchmarks: T2I-CompBench++ DPG-Bench TIFA

Metrics: T2I-CompBench++ category scores DPG-Bench global/entity/attribute/relation/other/all TIFA VQA score

Baselines: SEED-LLaMA DreamLLM Emu3 SD-v1.5 SD-v2 SDXL PixArt-alpha DALL-E 2 DALL-E 3

Builds on: DPO DreamLLM SEED-LLaMA MC Dropout RLAIF

Closest papers by meaning

• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.91)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.90)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.90)
• Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.89)
• A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.89)
• Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)