Research map / Vision-language and multimodal / Human and embodied modeling
Write and Paint: Generative Vision-Language Models are Unified Modal Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add prefix language modeling (PLM) objective (ablation) | DaVinci · COCO captioning, VQAv2, NLVR2, text-to-image | CIDEr / accuracy / FID: CIDEr 117.30 vs 111.17 without PLM; NLVR2 72.55 vs 53.28 | Helped |
| add prefix image modeling (PIM) objective (ablation) | DaVinci · NLVR2, ImageNet linear eval, text-to-image | accuracy / IS-FID: NLVR2 72.55 vs 69.78 without PIM; ImageNet 48.88 vs 45.54 | Helped |
| add text-to-text objective (ablation) | DaVinci · MNLI, ImageNet, COCO captioning | accuracy / CIDEr: MNLI 81.76 vs 76.48 without Text2Text; CIDEr 117.30 vs 115.21 | Helped |
| suffix-painting masking instead of random masked image modeling (ablation) | DaVinci image branch · COCO captioning, ImageNet, Food101, CIFAR10, text-to-image | CIDEr / accuracy / FID: CIDEr 117.3 vs 113.4 (MIM); ImageNet 48.88 vs 48.46; Food101 75.32 vs 56.95 | Helped |
| ResNet feature extraction instead of token projection (ablation) | DaVinci image encoder · COCO captioning, VQA, ImageNet | CIDEr / accuracy: CIDEr 117.3 vs 49.2 (token projection); VQA 69.25 vs 52.13 | Helped |
| ResNet feature extraction instead of ViT-style patch projection (ablation) | DaVinci image encoder · COCO captioning, ImageNet | CIDEr / accuracy: CIDEr 117.3 vs 79.5 (patch projection); ImageNet 48.88 vs 36.23 | Helped |
| dynamic masking ratio U(0,1) instead of static prefix length (0, 15%, 50%) (ablation) | DaVinci · COCO captioning, VQA, ImageNet, Food101, text-to-image | CIDEr / accuracy / FID: CIDEr 117.3 vs 117.0 (PL=0), 116.53 (PL=15%), 115.53 (PL=50%); ImageNet 48.88 vs 37.69 at PL=50% | Helped |
| DaVinci pretraining with prefix multimodal objectives vs SimVLM at 46.4M data | DaVinci vs SimVLM · GLUE/ImageNet linear eval/VQAv2 fine-tune | average and per-task scores: NLP avg 78.79 vs 78.79 (no gain); Vision avg 78.56 vs 74.14; VQAv2 73.89 vs 72.12 | Mixed |
| DaVinci at 647.7M data vs SimVLM at 647.7M data | DaVinci · Multi-modal Avg., zero-shot and fine-tune | average score: Multi-modal Avg. 62.50 vs 59.13 | Helped |
| DaVinci at 46.4M data vs FLAVA | DaVinci · GLUE and ImageNet linear eval average | average score: Compared with FLAVA, DaVinci improves an average of 2.04% (language) and 1.68% (vision) | Helped |
Closest papers by meaning
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024 · similarity 0.90)
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.89)
- GMAIL: Generative Modality Alignment for generated Image Learning (ICML 2025 · similarity 0.88)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.