Research map / Vision-language and multimodal / CLIP and zero-shot
Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Compression Learning (LCL), a vision encoder pre-training method that maximizes mutual information between inputs and outputs of a causal model over interleaved image-text data, combining contrastive and autoregressive losses to learn visual representations from scratch.
Problem
Vision model pre-training has relied on paired image-text data (CLIP), and no existing method can learn robust visual representations from scratch using interleaved image-text data, which is abundant on the web.
Approach
Latent compression of interleaved sequences: image latents from a ViT are fed with text into a causal LM; a contrastive loss aligns each image with its preceding context and an autoregressive loss predicts subsequent text, which the authors derive as maximizing mutual information and which avoids visual representation collapse.
Key results
- On MMC4 (frozen transfer, ViT-B), LCL reaches COCO captioning CIDEr 87.5 vs 82.9 for paired CLIP and 84.8 for CoCa, while matching classification (75.2 vs 74.8 CLIP).
- With ViT-L/14 pre-trained on LAION-400M+MMC4 (frozen transfer), LCL improves over OpenCLIP on LLaVA-1.5 multimodal benchmarks, e.g., VQAv2 70.7 vs 68.7 and MME 1291.7 vs 1266.2.
- Pre-training on LAION-Random interleaved data gives performance comparable to CLIP on paired data (ImageNet frozen 75.1 vs CLIP 75.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LCL instead of paired-data CoCa for MMC4 pre-training | ViT-L/14 vision pre-training · COCO captioning, frozen transfer on MMC4 | C: 84.8 -> 87.5 | Helped |
| Use LCL instead of paired-data CoCa for MMC4 pre-training | ViT-L/14 vision pre-training · ImageNet-1k, frozen transfer on MMC4 | acc-1: 75.4 -> 75.2 | Hurt |
| Use LCL instead of paired-data CoCa for LAION pre-training | ViT-L/14 vision pre-training · COCO retrieval, frozen transfer on LAION | TR@1: 48.6 -> 48.3 | Hurt |
| Add MMC4 to LAION pre-training data | LCL vision pre-training · VQAv2, frozen transfer | accuracy: 68.7 -> 70.7 | Helped |
| Add auto-regressive generation loss to contrastive loss (ablation) | LCL on MMC4 · COCO retrieval, frozen transfer | TR@1: 46.4 -> 48.5 | Helped |
| Add contrastive loss to generation loss (ablation) | LCL on MMC4 · COCO retrieval, frozen transfer | TR@1: 10.3 -> 48.5 | Helped |
| Set contrastive loss weight λ to 0.05 instead of 0.1 (ablation) | LCL on MMC4 · COCO retrieval, frozen transfer | TR@1: 48.5 -> 48.3 | Hurt |
| Set contrastive loss weight λ to 0.2 instead of 0.1 (ablation) | LCL on MMC4 · COCO retrieval, frozen transfer | TR@1: 48.5 -> 47.6 | Hurt |
| Set contrastive loss weight λ to 0.5 instead of 0.1 (ablation) | LCL on MMC4 · COCO retrieval, frozen transfer | TR@1: 48.5 -> 37.1 | Hurt |
| Add MMC4 to LAION pre-training data (ablation) | LCL on ViT-B/16 · COCO retrieval, frozen transfer | TR@1: 48.3 -> 49.4 | Helped |
Surprising findings
- Autoregressive-only text generation pre-training on MMC4 (Flamingo-style, Emu-style) collapses visual representations, reaching near-zero transfer performance despite similar text loss, suggesting models rely on redundant text context.
Limitations
- Gains over paired baselines on classification and retrieval are small or on par; LCL mainly helps captioning and multimodal dialogue.
- Contrastive weight lambda is sensitive (0.5 degrades results sharply), suggesting the two objectives can conflict.
- Evaluation of the text-generation side relies on frozen or full transfer with a small OPT-125M LM in pre-training and LLaVA-1.5 for dialogue, so scaling behavior is not established.
Experimental setup
Datasets: LAION-400M MMC4 OBELICS LAION-Random MMC4-Pair ImageNet-1K CC12M CC3M SBU MSCOCO Flickr30k LAION-COCO
Benchmarks: ImageNet-1K COCO retrieval Flickr30k retrieval COCO captioning NoCaps VQAv2 GQA VizWiz SQA POPE MME MMBench
Metrics: top-1 accuracy TR@1 IR@1 BLEU-4 CIDEr
Baselines: CLIP CoCa BLIP2 BEiT3 Flamingo Emu OpenCLIP
Builds on: CLIP OpenCLIP CoCa LLaVA-1.5 OPT
Closest papers by meaning
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.88)
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.88)
- Bridging Compressed Image Latents and Multimodal Large Language Models (ICLR 2025 · similarity 0.87)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.87)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.