Research map / Image generation / Text-to-image diffusion
TIPS: Text-Image Pretraining with Spatial awareness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIPS, a general-purpose image-text encoder that combines synthetic PaliGemma captions (dual [CLS] embedding with noisy web captions) with contrastive learning, self-distillation and masked image modeling to yield spatially aware frozen features for dense and global tasks.
Problem
Image-text contrastive models produce global features that transfer poorly to dense prediction tasks such as segmentation and depth estimation, where self-supervised image-only models remain preferred.
Approach
Replace or supplement noisy web captions with synthetic captions from a captioner, use a second [CLS] token with a dual CLIP loss to learn from both caption types, and add self-distillation and masked image modeling losses to CLIP so the patch features become spatially coherent.
Key results
- On ViT-B ablation, the full method improves PASCAL segmentation from 64.4 to 79.0 mIoU and reduces NYUv2 depth RMSE from 0.620 to 0.478 over the CLIP baseline.
- TIPS-g/14 HR reaches 83.6 mIoU on PASCAL and 49.9 on ADE20k, outperforming DINOv2-g/14 (83.0 and 49.0) on segmentation.
- On image-text retrieval, TIPS-g/14 HR attains 74.0 COCO I→T R@1 and 59.4 COCO T→I R@1, best among compared weakly-supervised models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace noisy captions with PaliGemma synthetic captions (ablation) | CLIP ViT-B · PASCAL VOC segmentation | mIoU: 64.4 -> 74.5 | Helped |
| replace noisy captions with PaliGemma synthetic captions (ablation) | CLIP ViT-B · ImageNet KNN classification | top-1 accuracy: 76.9 -> 70.0 | Hurt |
| use dual embeddings for noisy and synthetic captions (ablation) | CLIP ViT-B · PASCAL VOC segmentation | mIoU: 64.4 -> 73.3 | Helped |
| add self-distillation (ablation) | CLIP ViT-B with noisy captions · PASCAL VOC segmentation | mIoU: 64.4 -> 70.3 | Helped |
| add masked image modeling to CLIP + self-distillation (ablation) | CLIP ViT-B with noisy captions · PASCAL VOC segmentation | mIoU: +5.6 points | Helped |
| combine dual captions, self-distillation, and MIM (ablation) | CLIP ViT-B · Flickr I→T retrieval | R@1: +10.1 points | Helped |
| use the PaliGemma object-list caption (ablation) | CLIP ViT-B · PASCAL VOC segmentation | mIoU: 64.4 -> 73.8 | Helped |
| use the PaliGemma main-object caption (ablation) | CLIP ViT-B · PASCAL VOC segmentation | mIoU: 64.4 -> 61.9 | Hurt |
| curate WebLI rather than use raw or EN quality-filtered data (ablation) | CLIP ViT-B · UnED fine-grained retrieval | Recall@1: 45.8 -> 62.9 | Helped |
| use random masking at 75% rather than 50% or 25% (ablation) | TIPS ViT-B · NYUv2 depth | RMSE: 0.533 -> 0.478 | Helped |
| use blockwise rather than random masking (ablation) | TIPS ViT-B · NYUv2 depth | RMSE: 0.478 -> 0.491 | Hurt |
| add BYOL augmentations to local crops (ablation) | TIPS ViT-B · PASCAL VOC segmentation | mIoU: 79.0 -> 40.1 | Hurt |
Surprising findings
- Replacing web captions with synthetic captions improves dense segmentation by about 10 points but drops ImageNet KNN accuracy by 6.9 points, showing a dense/global trade-off that the dual embedding resolves.
- Even CLIP-L outperforms SigLIP-SO on all six dense evaluations, indicating that strong image-level performance does not transfer to dense features.
Limitations
- Relies on a synthetic captioner (PaliGemma) and LLM prompting (Gemini 1.5 Flash) for caption variants, adding cost and dependence on captioner quality.
- Training uses a proprietary-scale curated WebLI dataset, which is not released, limiting reproducibility of the data pipeline.
Experimental setup
Datasets: WebLI Mapillary SLS PASCAL VOC ADE20K NYUv2 ImageNet-1K ImageNet22k UnED DataComp
Benchmarks: PASCAL VOC ADE20K NYUv2 NAVI ImageNet-1K UnED Flickr30K COCO DOCCI Objaverse
Metrics: mIoU RMSE angular RMSE top-1 accuracy recall@1 PSNR
Baselines: CLIP OpenCLIP-G/14 SigLIP-SO/14 EVA-CLIP-g/14 SILC-G/16 DINOv2-g/14 DINO-B MAE-H/14 iBOT-L/16 JFT-3B-g/14 MaskCLIP-B/14
Builds on: CLIP DINO iBOT DINOv2 SILC MaskCLIP
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.88)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.88)
- Enhancing Vision-Language Pre-training with Rich Supervisions (CVPR 2024 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Textual Supervision Enhances Geospatial Representations in Vision-Language Models (ICML 2026 · similarity 0.87)
- Stitch and Tell: A Structured Data Augmentation Method for Spatial Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.