Research map / Vision-language and multimodal / CLIP and zero-shot
UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UMFC, a training-free and label-free calibration that removes domain-induced biases from CLIP image and text features using unlabeled multi-domain data, evaluated on DomainNet and ImageNet variants.
Problem
CLIP's zero-shot accuracy varies strongly across visual domains because its visual encoder encodes domain over category information and its text encoder favors domain-related class names; adapting to unlabeled multi-domain data without labels or training is the goal.
Approach
Cluster unlabeled image features into pseudo-domains, subtract per-cluster mean from image features (IFC), and estimate text-side domain bias by transferring the image-derived domain transition direction to text embeddings and subtracting it (TFC), averaging over clusters.
Key results
- On DomainNet under transductive learning, UMFC reaches 61.11% average accuracy vs. 57.88% for zero-shot CLIP, and UMFC+CLIP-E reaches 61.68% in unsupervised calibration.
- On quickdraw, UMFC improves CLIP from 14.23% to 19.67% in unsupervised calibration.
- On ImageNet-Variants under transductive learning, UMFC averages 63.17% vs. 61.22% for CLIP; UMFC+CLIP-E reaches 65.19%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply UMFC feature calibration | CLIP · DomainNet, transductive learning | accuracy (Avg): 57.88 -> 61.11 | Helped |
| Add CLIP-E ensemble prompts to UMFC | UMFC · DomainNet, unsupervised calibration | accuracy (Avg): 60.79 -> 61.68 | Helped |
| Add CLIP-E ensemble prompts to UMFC | UMFC · ImageNet-Variants, transductive learning | accuracy (Avg): 63.17 -> 65.19 | Helped |
| Use IFC (image feature calibration) (ablation) | CLIP · DomainNet, transductive learning | accuracy (Avg): 57.88 -> 60.85 | Helped |
| Use TFC (text feature calibration) (ablation) | CLIP · DomainNet, transductive learning | accuracy (Avg): 57.88 -> 58.14 | Helped |
| Combine IFC and TFC (ablation) | CLIP · DomainNet, transductive learning | accuracy (Avg): 60.85 -> 61.11 | Helped |
| Vary cluster number M from 3 to 10 (ablation) | UMFC · DomainNet, transductive learning | accuracy (Avg): 60.66; 61.59; 61.11; 61.71; 61.77 | Mixed |
| Vary test batch size from 1 to 100 (ablation) | UMFC · DomainNet, test-time adaptation | accuracy (Avg): 60.21; 60.44; 60.66; 60.76; 60.79; 60.85 | Mixed |
| Use EMA rather than storing batch features for calibration statistics (ablation) | UMFC · DomainNet, test-time adaptation | accuracy (Avg): 60.85 vs 60.57 | No clear effect |
| Calibrate using single-domain unlabeled samples instead of multi-domain samples (ablation) | UMFC · DomainNet, unsupervised calibration (Avg) | accuracy: 59.85; 58.72; 59.73 | Mixed |
Surprising findings
- Training-free UMFC outperforms MUST, which fine-tunes CLIP on the same unlabeled multi-domain data, on average DomainNet accuracy (60.79 vs. 60.89 is roughly tied, but UMFC beats MUST on most domains excluding clipart and sketch).
- Few-shot CoOp trained on a single domain improves its own domain but degrades on other domains, e.g., CoOp (Q) drops average accuracy to 36.39%.
Limitations
- Requires unlabeled samples from the target domains to estimate calibration vectors.
- Relies on clustering to recover domain structure; performs less well on ImageNet-A, which lacks distinct domain styles.
- Based on a fixed prompt; gains on ImageNet-Variants are smaller than on DomainNet in some cases.
Experimental setup
Datasets: DomainNet ImageNet-A ImageNet-R ImageNet-Sketch
Benchmarks: DomainNet ImageNet-Variants (IN-A, IN-R, IN-S)
Metrics: accuracy
Baselines: CLIP CLIP-E CLIP-D MUST CoOp CLIP-Adapter MIRO PromptStyler TPT
Builds on: CLIP Test-Time Prompt Tuning (TPT) MUST
Closest papers by meaning
- Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.90)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.89)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
- An Empirical Study Into What Matters for Calibrating Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.