Research map / Vision-language and multimodal / CLIP and zero-shot
FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLAME, which uses a frozen LLM (Mistral-Nemo) as the text encoder for CLIP-style pre-training, extracting multiple facet embeddings per long caption via multifaceted prompts, with facet-decoupled attention and offline embedding caching.
Problem
CLIP-style language-image pre-training is data-hungry and its text encoder is limited to 77 tokens, which hurts long-form captions, multilingual generalization and data efficiency.
Approach
Replace the CLIP text encoder with a frozen LLM and extract several semantic facet embeddings per caption using multifaceted prompts; a facet-decoupled attention mask lets all prompts share one forward pass, and the frozen LLM embeddings are precomputed offline, so only the visual encoder and a projection MLP are trained.
Key results
- On CC3M, FLAME improves ImageNet zero-shot top-1 by 4.9% over DreamLIP (36.0 vs 31.1).
- On YFCC15M, FLAME averages 51.7% image-to-text and 49.0% text-to-image recall@1 on Crossmodal-3600 across 36 languages, 44.4% and 44.5% above WIT-400M CLIP.
- Trained on YFCC15M, FLAME reaches 87.9% text-to-image recall@1 on Urban-1k, 34.6% above WIT-400M CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLAME with frozen Mistral-Nemo and 7 semantic prompts | ViT-B/16 language-image pre-training on CC3M · ImageNet zero-shot | top-1 accuracy: 31.1 -> 36.0 | Helped |
| Use FLAME with frozen Mistral-Nemo and 7 semantic prompts | ViT-B/16 language-image pre-training on YFCC15M · ImageNet zero-shot | top-1 accuracy: 48.2 -> 51.5 | Helped |
| Use FLAME instead of WIT-400M-trained CLIP | YFCC15M-trained language-image model · Urban-1k text-to-image retrieval | recall@1: 53.3 -> 87.9 | Helped |
| Use FLAME instead of DreamLIP | YFCC15M-trained language-image model · Flickr30k image retrieval | R@1: 65.3 -> 73.3 | Helped |
| Use FLAME instead of CLIP | YFCC15M-trained model evaluated on multilingual ImageNet · ImageNet multilingual average | top-1 accuracy: 19.2 -> 40.6 | Helped |
| Use 7 prompts instead of 1 (ablation) | FLAME · ImageNet zero-shot, CC3M | top-1 accuracy: 33.8 -> 36.0 | Helped |
| Use 7 prompts instead of 3 (ablation) | FLAME · ImageNet zero-shot, CC3M | top-1 accuracy: 34.1 -> 36.0 | Helped |
| Use 7 prompts instead of 5 (ablation) | FLAME · ImageNet zero-shot, CC3M | top-1 accuracy: 35.2 -> 36.0 | Helped |
| Use 9 prompts instead of 7 (ablation) | FLAME · ImageNet zero-shot, CC3M | top-1 accuracy: 36.0 -> 35.6 | Hurt |
| Replace Mistral-Nemo with Mistral-7B (ablation) | FLAME with ViT-B/16 · ImageNet zero-shot | top-1 accuracy: 36.0 -> 34.4 | Hurt |
| Replace Mistral-Nemo with Llama-3.1-8B (ablation) | FLAME with ViT-B/16 · ImageNet zero-shot | top-1 accuracy: 36.0 -> 34.1 | Hurt |
| Replace ViT-B/16 with ViT-S/16 (ablation) | FLAME with Mistral-Nemo · ImageNet zero-shot | top-1 accuracy: 36.0 -> 32.4 | Hurt |
Surprising findings
- Training only on English YFCC15M, FLAME generalizes to 36 languages and beats mSigLIP trained on multilingual WebLI.
- The frozen LLM is used as text encoder despite prior conventional wisdom that frozen text encoders underperform.
Limitations
- Extra computational overhead during initial offline text embedding extraction.
- Multiple prompts per caption require several embeddings per sample.
Experimental setup
Datasets: CC3M YFCC15M ShareGPT4V Crossmodal-3600 Urban-1k MSCOCO Flickr30k ImageNet DOCCI
Benchmarks: Urban-1k ShareGPT4V-val DCI DOCCI-test Crossmodal-3600 MSCOCO Flickr30k ImageNet Food-101 CIFAR-10 CIFAR-100 SUN397
Metrics: recall@1 top-1 accuracy average accuracy
Baselines: CLIP Long-CLIP DreamLIP LaCLIP MLLM-A CN-CLIP NLLB-CLIP mSigLIP HiCLIP SoftCLIP
Builds on: CLIP DreamLIP Long-CLIP OpenCLIP
Closest papers by meaning
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.88)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.88)
- LoTLIP: Improving Language-Image Pre-training for Long Text Understanding (NeurIPS 2024 · similarity 0.88)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.