Research map / Vision-language and multimodal / Human and embodied modeling
A Vision Check-up for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates whether text-only LLMs (GPT-3.5, GPT-4, Davinci) can draw, recognize, and self-correct visual concepts via code, and uses LLM-generated code-rendered images to pretrain a ResNet-50 with MoCo-v2 that transfers to ImageNet.
Problem
Whether language models, trained only on text, acquire usable knowledge of the visual world, and whether their code-rendered images can serve as training data for vision systems on natural images.
Approach
Represent images as rendering code so text-only LLMs can generate, recognize, and iteratively self-correct visual concepts across a hierarchy (shapes, objects, scenes), then use the generated images as a procedural pretraining source for self-supervised visual representation learning.
Key results
- Self-feedback lifts GPT-3.5 median CLIP percentile so that after ~20 feedback rounds it approaches GPT-4 without feedback.
- ResNet-50 pretrained on LLM-generated images mixed with Shaders-21k reaches 46.03% top-1 linear ImageNet-1k accuracy, above Shaders-21k alone (44.83%).
- On recognizing human drawings as code, GPT-3.5 scores 0.380 on scenes versus 0.2 chance, while Davinci and GPT-4 are near chance on objects.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-generated textual feedback over repeated correction rounds | GPT-3.5 image generation · Visual Aptitude Dataset | median CLIP percentile | Helped |
| Train on LLM-generated images together with S-21k textures | ResNet-50 pretraining · ImageNet-1k | top-1 linear accuracy: 44.83 -> 46.03 | Helped |
| Train on LLM-generated images together with S-21k textures | ResNet-50 pretraining · ImageNet-100 | 5-NN accuracy: 43.24 -> 43.36 | Helped |
| Exclude GPT-4 images from LLM-generated pretraining data (ablation) | ResNet-50 pretrained on LLM images and S-21k · ImageNet-1k | top-1 linear accuracy: 46.03 -> 45.79 | No clear effect |
| Exclude GPT-4 images from LLM-generated pretraining data (ablation) | ResNet-50 pretrained on LLM images and S-21k · ImageNet-100 | 5-NN accuracy: 43.36 -> 43.40 | No clear effect |
| Exclude GPT-4 images from LLM-generated pretraining data (ablation) | ResNet-50 pretrained on LLM images · ImageNet-1k | top-1 linear accuracy: 36.16 -> 33.60 | No clear effect |
| Exclude GPT-4 images from LLM-generated pretraining data (ablation) | ResNet-50 pretrained on LLM images · ImageNet-100 | 5-NN accuracy: 27.44 -> 22.42 | No clear effect |
| Use GPT-3.5 to recognize human-drawn scenes rather than chance classification | GPT-3.5 · Human drawings, scenes | accuracy: 0.2 -> 0.380 | Helped |
| Use GPT-4 to recognize human-drawn objects rather than chance classification | GPT-4 · Human drawings, objects | accuracy: 0.2 -> 0.234 | Helped |
Surprising findings
- Models can fail to recognize concepts they can render well, contradicting the notion that verification is easier than creation.
- LLM-generated images alone (without textures) underperform simple procedural datasets like FractalDB, and the GPT-4 subset adds little over the set without it.
Limitations
- LLM-generated images look unnatural and lack texture, limiting pretraining quality.
- Models struggle with digits, precise shapes, textures, and spatial organization.
- Recognition performance on human-drawn code is only barely above chance for most models.
Experimental setup
Datasets: ImageNet-1k ImageNet-100 ADE20K MS-COCO Shaders-21k Dead-leaves FractalDB StyleGAN procedural data Places
Benchmarks: Visual Aptitude Dataset ImageNet-1k ImageNet-100
Metrics: CLIP retrieval percentile (fidelity) LPIPS diversity FID top-1 linear accuracy 5-NN accuracy classification accuracy on human drawings
Baselines: Stable Diffusion Dead-leaves FractalDB-1k StyleGAN procedural data Shaders-21k Places (real images) Random init
Builds on: MoCo-v2 Learning to see by looking at noise / dead-leaves pretraining FractalDB Shaders-21k MixUp
Closest papers by meaning
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.91)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.90)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.