Research map / Vision-language and multimodal / Multimodal LLMs
LLMGA: Multimodal Large Language Model based Generation Assistant
bin xia, Shiyin Wang, Yingfan Tao, Yitong Wang, Jiaya Jia
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune SD1.5 in the second stage to follow detailed MLLM prompts (ablation) | LLMGA-7b · MSCOCO T2I | FID: 23.5946 -> 18.5234 | Improved |
| add second-stage SD finetuning (ablation) | LLMGA-7b · Places outpainting (wide masks) | FID: 3.2630 -> 2.4409 | Improved |
| use LLMGA-7B prompts instead of plain SD1.5 | SD1.5 · MSCOCO T2I | FID: 24.0081 -> 18.5234 | Improved |
| scale MLLM from 7B to 13B (ablation) | LLMGA · MSCOCO T2I | FID: 18.5234 -> 18.4063 | Improved |
| add reference input (masked image) to restoration network (ablation) | DiffIR on LLMGA outpainting · Places outpainting | FID: 2.3587 -> 2.2993 | Improved |
| add random brightness/contrast degradation in restoration training (ablation) | DiffRIR · Places outpainting | FID: 2.2993 -> 2.2687 | Improved |
| control SD with detailed language prompt instead of fixed-size embedding (ablation) | LLMGA · MMBench | accuracy: 35.3 -> 62.9 | Improved |
| control SD with language prompt instead of embedding over multiple conversation turns | LLMGA · MSCOCO multi-turn generation | image quality | Improved |
| train with full prompt refinement data instead of 10% (ablation) | LLMGA · MSCOCO T2I | FID: 21.4460 -> 18.5234 | Improved |
| train with full inpainting & outpainting data instead of 10% (ablation) | LLMGA · Places outpainting | FID: 3.1054 -> 2.4409 | Improved |
| train with full similar image generation data instead of 10% (ablation) | LLMGA · MSCOCO T2I | FID: 19.5698 -> 18.5234 | Improved |
| use LLMGA zero-shot for editing instead of training SD on editing data (vs MagicBrush) | LLMGA · MagicBrush test | L1: 0.0647 (MagicBrush) vs 0.0814 (LLMGA) | Mixed |
Closest papers by meaning
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.91)
- GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing (NeurIPS 2024 · similarity 0.90)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.90)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
