Research map / Vision-language and multimodal / CLIP and zero-shot
un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves CLIP's visual detail capture by finetuning the CLIP image encoder while freezing a pretrained unCLIP (Stable unCLIP) generator that is conditioned on CLIP image embeddings, using only image data (CC3M).
Problem
CLIP's global image-text contrastive training leaves its image encoder weak at capturing fine visual details, hurting dense prediction and vision-centric MLLM tasks; collecting region-text data or retraining CLIP is costly.
Approach
Invert unCLIP: finetune the CLIP image encoder by maximizing the likelihood of images under a frozen unCLIP diffusion generator conditioned on the CLIP image embedding. Because the generator's input lives in CLIP's embedding space, no trainable projector is needed and language alignment is preserved.
Key results
- On MMVP-VLM, ours raises OpenAI ViT-L-14@224 average from 19.3 to 32.6 and SigLIP ViT-SO-14@384 from 37.0 to 41.5, beating DIVA and matching GenHancer.
- Replacing CLIP with the finetuned encoder in ClearCLIP improves open-vocabulary segmentation average mIoU from 30.8 to 34.3 across eight datasets.
- On zero-shot ImageNet-1K classification the finetuned model drops from 75.5 to 62.4, reflecting a trade-off away from coarse classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune the CLIP image encoder by inverting unCLIP | OpenAI ViT-L-14@224 · MMVP-VLM | Avg: 19.3 -> 32.6 | Helped |
| finetune the CLIP image encoder by inverting unCLIP | OpenAI ViT-L-14@336 · MMVP-VLM | Avg: 20.0 -> 30.4 | Helped |
| finetune the CLIP image encoder by inverting unCLIP | OpenCLIP ViT-H-14@224 · MMVP-VLM | Avg: 28.9 -> 36.3 | Helped |
| finetune the CLIP image encoder by inverting unCLIP | SigLIP ViT-SO-14@384 · MMVP-VLM | Avg: 37.0 -> 41.5 | Helped |
| replace the original CLIP encoder with un²CLIP | CLIP segmentation inference · Open-vocabulary segmentation across CLIP, MaskCLIP, SCLIP, and ClearCLIP | Average mIoU: 4.3 -> 6.0; 12.7 -> 15.3; 16.1 -> 23.0; 30.8 -> 34.3 | Helped |
| use un²CLIP as the LLaVA vision encoder | LLaVA-1.5 with Vicuna-7B · MMVP | score: 24.7 -> 31.3 | Helped |
| use un²CLIP as the LLaVA vision encoder | LLaVA-1.5 with Vicuna-7B · NaturalBench | Acc: 67.3 -> 68.7 | Helped |
| use un²CLIP as the LLaVA vision encoder | LLaVA-1.5 with Vicuna-7B · CV-Bench 2D ADE20K | score: 49.6 -> 53.9 | Helped |
| use un²CLIP as the LLaVA vision encoder | LLaVA-1.5 with Vicuna-7B · POPE rand | score: 87.3 -> 88.0 | Helped |
| use un²CLIP as the LLaVA vision encoder | LLaVA-1.5 with Vicuna-7B · Hallusion Avg. | score: 27.6 -> 28.4 | Mixed |
| add a randomly initialized projection layer (ablation) | un²CLIP with OpenAI ViT-L-14@224 · MMVP-VLM | MMVP-VLM: 32.6 -> 16.3 | Hurt |
| add an identity-initialized projection layer (ablation) | un²CLIP with OpenAI ViT-L-14@224 · MMVP-VLM | MMVP-VLM: 32.6 -> 30.4 | Hurt |
Surprising findings
- Zero-shot classification drops substantially (ImageNet-1K 75.5 to 62.4), indicating a trade-off between visual detail and coarse semantics.
- Adding a trainable projection layer or unfreezing the generator lowers recognition performance despite improving diffusion loss.
Limitations
- Requires a pretrained unCLIP model aligned with the CLIP backbone; unCLIP training is costly and only exists for some CLIP backbones.
- Zero-shot classification and retrieval degrade on some benchmarks (e.g., ImageNet-1K drops from 75.5 to 62.4).
- MLLM experiments only use the 7B Vicuna LLaVA-1.5 setup due to compute constraints.
Experimental setup
Datasets: CC3M
Benchmarks: MMVP-VLM ADE20K PASCAL VOC PASCAL Context COCO-Stuff Cityscapes COCO Object MMVP NaturalBench CV-Bench POPE SciQA-IMG
Metrics: accuracy mIoU diffusion loss recall@5
Baselines: DIVA GenHancer Original CLIP MaskCLIP SCLIP ClearCLIP
Builds on: unCLIP Stable Diffusion DIVA CLIP ClearCLIP
Closest papers by meaning
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.92)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.91)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.90)
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing (ICLR 2026 · similarity 0.90)
- TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives (NeurIPS 2024 · similarity 0.89)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.