Research map / Vision-language and multimodal / Human and embodied modeling
LoTLIP: Improving Language-Image Pre-training for Long Text Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-captions 100M web images with long MLLM captions and adds learnable corner tokens with an attention mask to a LiT-style BERT text encoder, so the model improves long-text retrieval while keeping short-text and classification performance.
Problem
Language-image pre-training models trained on short captions struggle to understand long texts, with salient tokens overshadowing others, and training on long captions can degrade short-text understanding such as ImageNet classification.
Approach
Pre-train with long re-captions alongside short captions, and insert learnable corner tokens into the text encoder with an attention mask so that corner tokens and the CLS token aggregate diverse sub-caption information through separate contrastive losses.
Key results
- On long-text-image retrieval, LoTLIP trained on 100M data reaches 83.72% average R@1 versus 79.52% for Long-CLIP (400M+1M), a 4.2-point gain.
- At 3M scale with ViT-B/16, LoTLIP improves ImageNet zero-shot accuracy from 43.76% (LiT) to 50.34%, and MSCOCO I2T R@1 from 34.20% to 46.56%.
- Long-text training with LiT alone raises long-text retrieval but drops ImageNet accuracy to 48.87% from 43.76%, showing short-text recovery needs corner tokens; LoTLIP recovers to 50.34%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add long texts to LiT pre-training (ablation) | LiT · DCI | R@1 I2T: 27.14 -> 47.96 | Helped |
| Add long texts to LiT pre-training (ablation) | LiT · ImageNet1k | Acc@1: 43.76 -> 48.87 | Helped |
| Add long texts and corner tokens | LiT · DCI | R@1 I2T: 27.14 -> 49.46 | Helped |
| Remove corner tokens and attention mask (ablation) | LoTLIP · DCI | R@1 I2T: 49.46 -> 47.96 | Hurt |
| Remove the attention mask (ablation) | LoTLIP with 2 corner tokens · DCI | R@1 I2T: 49.46 -> 48.61 | Hurt |
| Increase corner-token count from 2 to 4 (ablation) | LoTLIP · MSCOCO | R@1 I2T: 46.56 -> 47.70 | Helped |
| Replace corner tokens with register tokens (ablation) | LoTLIP · DCI | R@1 I2T: 49.46 -> 45.62 | Hurt |
| Increase the number of long-text sub-captions (ablation) | LiT and LoTLIP · Long-text-image retrieval | R@1 | Helped |
| Increase the number of long-text sub-captions (ablation) | LiT and LoTLIP · MSCOCO and ImageNet | R@1 / Acc@1 | Hurt |
| Use long captions from all three MLLMs instead of one source (ablation) | LoTLIP · Downstream tasks | Average performance | Helped |
| Train with long captions generated by ShareGPT4V (ablation) | LoTLIP · Long-text-image retrieval | R@1 | Helped |
Surprising findings
- Adding register tokens to the text encoder did not improve performance, while corner tokens did.
- Long-CLIP-style component matching applied to LiT reduced ImageNet accuracy relative to LiT (46.82% vs 48.87% in the 3M ablation).
Limitations
- Synthetic long captions contain hallucinations that may limit gains.
- Gains beyond about 3 sub-captions plateau while training cost grows, with a clear performance-efficiency trade-off.
- Image encoder is frozen and initialized from pre-trained weights, so results depend on LiT setup.
Experimental setup
Datasets: CC3M CC12M YFCC15M LAION COYO DCI IIW ShareGPT4V MSCOCO Flickr30k ImageNet
Benchmarks: DCI IIW ShareGPT4V-1k ShareGPT4V-10k MSCOCO Flickr30k ImageNet
Metrics: R@1 R@5 Acc@1
Baselines: LiT Long-CLIP LiT+Long-CLIP FILIP LaCLIP SigLIP CLIP ALIGN
Builds on: LiT CLIP Long-CLIP DreamLIP
Closest papers by meaning
- Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation (CVPR 2025 · similarity 0.90)
- TULIP: Token-length Upgraded CLIP (ICLR 2025 · similarity 0.90)
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.89)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.88)
- FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-training (CVPR 2025 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.