Research map / Vision-language and multimodal / CLIP and zero-shot
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COSMIC, a training-free CLIP test-time adaptation method that builds a cache with text, CLIP and DINOv2 features as graphs and queries it via maximal-clique hyper-class centers.
Problem
Cache-based test-time adaptation for CLIP suffers from noisy pseudo-labels entering the cache and from querying with single-class centers, which hurts accuracy under distribution shift.
Approach
Builds dual graphs (CLIP shared semantic space with text and visual centers, plus an auxiliary DINOv2 fine-grained visual space) and uses maximal cliques as hyper-class centers to select correlated inlier classes for adaptive, training-free prediction.
Key results
- On OOD benchmarks with CLIP ViT-B/16, COSMIC reaches 72.83% OOD average vs 63.80% for DMN (+9.03%).
- On 10 cross-domain datasets with CLIP RN-50, COSMIC averages 68.65% vs 63.32% for DMN (about +5.33%).
- On Flower102 with CLIP ViT-B/16, COSMIC improves over CLIP by 15.47% using 16 views, versus 7.50% for DMN with 128 views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace DMN with COSMIC | CLIP-RN-50 · ImageNet OOD average | top-1 accuracy (%): 46.49 -> 62.30±0.05 | Helped |
| Replace TDA with COSMIC | CLIP-ViT-B/16 · ImageNet OOD average | top-1 accuracy (%): 63.89 -> 72.83±0.10 | Helped |
| Replace DMN with COSMIC | CLIP-RN-50 · 10 cross-domain datasets average | top-1 accuracy (%): 63.32 -> 68.65±0.32 | Helped |
| Replace DMN with COSMIC | CLIP-ViT-B/16 · 10 cross-domain datasets average | top-1 accuracy (%): 70.30 -> 72.81±0.09 | Helped |
| Add CSS graph prediction to CLIP (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · ImageNet-Val | top-1 accuracy (%): 69.91 -> 71.88; 1.97 | Helped |
| Add AFV graph prediction to CLIP (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · ImageNet-Val | top-1 accuracy (%): 69.91 -> 67.05; -2.86 | Hurt |
| Use both CSS and AFV graph predictions (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · ImageNet-Val | top-1 accuracy (%): 69.91 -> 73.79; 3.88 | Helped |
| Add CSS graph prediction to CLIP (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · Flower102 | top-1 accuracy (%): 72.64 -> 74.46; 1.82 | Helped |
| Add AFV graph prediction to CLIP (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · Flower102 | top-1 accuracy (%): 72.64 -> 77.71; 5.07 | Helped |
| Use both CSS and AFV graph predictions (ablation) | CLIP ViT-B/16 with DINOv2 ViT-S/14 · Flower102 | top-1 accuracy (%): 72.64 -> 81.61; 8.97 | Helped |
| Use DINOv2 ViT-S/14 instead of no auxiliary encoder (ablation) | CLIP ViT-B/16 · ImageNet-Val | top-1 accuracy (%): 69.91 -> 73.79; 3.88 | Helped |
| Use DINOv2 ViT-B/14 instead of no auxiliary encoder (ablation) | CLIP ViT-B/16 · ImageNet-Val | top-1 accuracy (%): 69.91 -> 76.73; 6.82 | Helped |
Surprising findings
- The auxiliary AFV graph alone degrades ImageNet-Val accuracy by 2.86% relative to CLIP, though it helps when fused with CSS.
Limitations
- Maximal clique search has non-trivial worst-case time complexity, adding computational burden.
- AFV graph alone can hurt accuracy on some datasets (e.g., ImageNet-Val) because of domain gaps in cached DINOv2 features.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Benchmarks: ImageNet ImageNet-A ImageNet-V2 ImageNet-R ImageNet-Sketch Aircraft Caltech101 Cars DTD EuroSAT Flower102 Food101
Metrics: top-1 accuracy average accuracy inference time per image
Baselines: CoOp CoCoOp TPT DiffTPT TDA DMN
Builds on: TDA DMN Tip-Adapter Bron-Kerbosch algorithm
Closest papers by meaning
- Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation (ICLR 2025 · similarity 0.87)
- What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective (NeurIPS 2026 · similarity 0.87)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.86)
- Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.86)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
- SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.