Research map / Vision-language and multimodal / Human and embodied modeling
TULIP: Token-length Upgraded CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Upgrades CLIP text encoders from absolute to RoPE relative position encodings via cosine distillation on the first 77 tokens, then NTK-scaled RoPE fine-tuning on long captions, enabling long-caption retrieval and text-to-image generation.
Problem
CLIP text encoders use fixed absolute positional encodings capped at 77 tokens, so long captions are truncated and long-caption understanding suffers; retraining with flexible encodings from scratch is costly.
Approach
Swap absolute positional encodings for RoPE in the CLIP text encoder, distill the original CLIP text encoder into this student on 77-token captions using cosine loss, then fine-tune with NTK-aware scaled RoPE on full-length captions jointly with short-caption contrastive loss.
Key results
- On Long-DCI with ViT-L-14, TULIP reaches 55.7 Img2Txt and 56.4 Txt2Img R@1 versus 54.0 / 46.1 for Long-CLIP.
- On Urban-1K with ViT-L-14, TULIP reports 90.1 Img2Txt and 91.1 Txt2Img R@1, above Long-CLIP's 82.7 / 86.1.
- In a 15-annotator human study on 20 prompts, SDXL with TULIP was preferred 89% of the time versus 11% for SDXL with CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace stretched absolute positional encodings with relative RoPE and adapt for long captions | CLIP ViT-L-14 · Long-DCI, ShareGPT4V, and Urban-1K long-caption retrieval | recall: 54.0 -> 55.7; 46.1 -> 56.4; 95.8 -> 99.0; 95.6 -> 99.0; 82.7 -> 90.1; 86.1 -> 91.1 | Helped |
| Replace stretched absolute positional encodings with relative RoPE and adapt for long captions | CLIP ViT-L-14 · COCO and Flickr30k short-caption retrieval | R@1 and R@5: 62.8 -> 62.6; 85.1 -> 84.7; 46.3 -> 46.1; 70.8 -> 71.1; 53.4 -> 56.7; 77.5 -> 79.5; 41.2 -> 41.6; 64.1 -> 64.3 | Mixed |
| Use RoPE instead of absolute positional encodings (ablation) | TULIP · Long-DCI retrieval | recall: 41.9 -> 55.7; 40.0 -> 56.4 | Helped |
| Use RoPE instead of CoPE (ablation) | TULIP · Long-DCI and Urban-1K retrieval | recall: 50.8 -> 55.7; 49.9 -> 56.4; 86.7 -> 90.1; 82.8 -> 91.1 | Helped |
| Increase context length from 77 to 154 tokens (ablation) | TULIP with RoPE · ShareGPT4V, Urban-1K, and Long-DCI retrieval | retrieval performance | Helped |
| Increase context length to 308 tokens (ablation) | TULIP with RoPE · Long-caption retrieval | retrieval performance | Mixed |
| Use cosine distillation loss instead of L2 (ablation) | TULIP distilled text encoder · Long-DCI, ShareGPT4V, and Urban-1K retrieval before expansion | recall: 39.4 -> 38.5; 35.8 -> 35.8; 84.8 -> 84.8; 83.9 -> 84.2; 70.2 -> 73.6; 56.3 -> 56.6 | Mixed |
| Use cosine distillation loss instead of MSE (ablation) | TULIP distilled text encoder · Long-DCI, ShareGPT4V, and Urban-1K retrieval before expansion | recall: 36.4 -> 38.5; 31.8 -> 35.8; 83.0 -> 84.8; 81.3 -> 84.2; 74.0 -> 73.6; 53.9 -> 56.6 | Helped |
| Increase student-model training data from 33% to 100% (ablation) | TULIP student model · Long-DCI, ShareGPT4V, and Urban-1K retrieval | top R@1: 15.2 -> 55.7; 12.2 -> 56.4; 88.6 -> 99.0; 90.0 -> 99.0; 33.6 -> 90.1; 36.9 -> 91.1 | Helped |
| Increase student-model training data from 66% to 100% (ablation) | TULIP student model · Long-DCI, ShareGPT4V, and Urban-1K retrieval | top R@1: 28.7 -> 55.7; 26.2 -> 56.4; 94.6 -> 99.0; 95.5 -> 99.0; 37.6 -> 90.1; 45.0 -> 91.1 | Helped |
Surprising findings
- Fine-tuned CLIP on ShareGPT4V degrades short-caption COCO retrieval (R@1 37.4 vs 51.8 for CLIP ViT-B-16).
- Long-CLIP drops sharply on ARO COCO and Flickr (23.3 and 24.8) compared with CLIP (47.9 and 60.2).
Limitations
- Performance is bounded by the quality of GPT-4V-generated ShareGPT4V captions.
- Token length is effectively constrained by the average caption length in ShareGPT4V (~174 tokens), so gains plateau beyond about 154-231 tokens.
- On short-caption COCO and Flickr30k retrieval, TULIP is only competitive with Long-CLIP and not uniformly better.
Experimental setup
Datasets: ShareGPT4V COCO Flickr30k Dense Captioning Images (DCI) DOCCI ARO VL-Checklist
Benchmarks: Long-DCI ShareGPT4V test Urban-1K COCO 5k Flickr30k DOCCI ARO VL-Checklist
Metrics: recall@1 recall@5 win rate
Baselines: CLIP Fine-tuned CLIP Long-CLIP CoPE PIXART-Alpha ELLA
Builds on: CLIP RoPE NTK-aware scaled RoPE Long-CLIP
Closest papers by meaning
- FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs (CVPR 2025 · similarity 0.91)
- LoTLIP: Improving Language-Image Pre-training for Long Text Understanding (NeurIPS 2024 · similarity 0.90)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.88)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.88)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.88)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.