Research map / Vision-language and multimodal / CLIP and zero-shot
Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongD-CLIP, a dual-teacher distillation framework that fine-tunes CLIP on long captions and distills it with the original CLIP as a second teacher to extend text input to 248 tokens while limiting knowledge forgetting.
Problem
CLIP's 77-token text limit and short-caption training hurt long-text retrieval, and fine-tuning for long text causes forgetting of zero-shot classification ability.
Approach
A long-text fine-tuned CLIP teacher and the original CLIP jointly teach a student: the text encoder is distilled from both (short and long text), and the image encoder is distilled from both with a dimension-wise importance re-weighting, preserving zero-shot knowledge while improving long-text representation.
Key results
- On ShareGPT4V image-to-text retrieval with ViT-B/16, LongD-CLIP reaches R@1 99.0 vs Long-CLIP 94.6; on Urban-1k image-to-text with ViT-L/14 it reaches 91.9 vs 82.7.
- Zero-shot classification average over five datasets with ViT-L/14 is 70.56, above CLIP (69.92) and Long-CLIP (69.78).
- Ablation shows the long-text teacher alone drops zero-shot average to 67.86, while the distilled student recovers it to 70.56.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dual-teacher distillation instead of Long-CLIP | CLIP ViT-L/14 · Urban-1k image-to-text retrieval | R@1: 82.7 -> 91.9 | Helped |
| use dual-teacher distillation instead of Long-CLIP | CLIP ViT-L/14 · ShareGPT4V image-to-text retrieval | R@1: 95.8 -> 98.3 | Helped |
| use dual-teacher distillation instead of Long-CLIP | CLIP ViT-L/14 · Flickr30k image-to-text retrieval | R@1: 53.4 -> 60.2 | Helped |
| use dual-teacher distillation instead of CLIP | CLIP ViT-L/14 · zero-shot classification average | accuracy: 69.92 -> 70.56 | Helped |
| train with the long-text teacher plus contrastive learning and an original-CLIP embedding constraint (ablation) | CLIP ViT-L/14 · ShareGPT4V image-to-text retrieval | R@1: 81.8 -> 97.3 | Helped |
| train with the long-text teacher plus contrastive learning and an original-CLIP embedding constraint (ablation) | CLIP ViT-L/14 · zero-shot classification average | accuracy: 69.92 -> 69.94 | No clear effect |
| use the long-text teacher alone instead of dual-teacher distillation (ablation) | CLIP ViT-L/14 · ShareGPT4V image-to-text retrieval | R@1: 81.8 -> 99.2 | Helped |
| use the long-text teacher alone instead of dual-teacher distillation (ablation) | CLIP ViT-L/14 · zero-shot classification average | accuracy: 69.92 -> 67.86 | Hurt |
| use MSE loss only for image-encoder distillation (ablation) | CLIP ViT-L/14 · ShareGPT4V image-to-text retrieval | R@1: 96.0 vs 98.3 | Hurt |
| use cosine similarity loss only for image-encoder distillation (ablation) | CLIP ViT-L/14 · ShareGPT4V image-to-text retrieval | R@1: 97.2 vs 98.3 | Hurt |
| use MSE loss only for image-encoder distillation (ablation) | CLIP ViT-L/14 · Urban-1k image-to-text retrieval | R@1: 83.4 vs 91.9 | Hurt |
| use cosine similarity loss only for image-encoder distillation (ablation) | CLIP ViT-L/14 · Urban-1k image-to-text retrieval | R@1: 88.3 vs 91.9 | Hurt |
Surprising findings
- The long-text teacher alone loses about 2 points of zero-shot classification accuracy, showing fine-tuning causes forgetting.
- Contrastive learning with an embedding constraint to the original CLIP (O-teacher + CL) hurts long-text representation learning.
Limitations
- Evaluation uses only ShareGPT4V and Urban-1k for long text, with Urban-1k captions generated by GPT-4V.
- The 'Ours' model is not uniformly best; e.g., ViT-L/14 COCO I2T and ImageNet-O results are mixed, and the L-teacher is better on some long-text metrics.
- Hardware and compute details are not reported.
Experimental setup
Datasets: ShareGPT4V
Benchmarks: ShareGPT4V validation Urban-1k COCO2017 5k Flickr30k ImageNet-1K ImageNet-V2 ImageNet-O CIFAR-10 CIFAR-100
Metrics: R@1 R@5 R@10 zero-shot accuracy CLIP Score
Baselines: CLIP Long-CLIP O-teacher + CL L-teacher
Builds on: CLIP Long-CLIP knowledge distillation
Closest papers by meaning
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.90)
- LoTLIP: Improving Language-Image Pre-training for Long Text Understanding (NeurIPS 2024 · similarity 0.90)
- GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.89)
- Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions (NeurIPS 2025 · similarity 0.88)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.