Research map / Vision-language and multimodal / CLIP and zero-shot
DiffCLIP: Few-shot Language-driven Multimodal Classifier
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DiffCLIP pretrains a modality-shared ViT encoder with unsupervised mask diffusion on unlabeled multimodal remote sensing patches, then fine-tunes it with CLIP-style language-driven contrastive learning for few-shot classification.
Problem
CLIP transfers poorly to high-dimensional multimodal remote sensing imagery, and image-text pairs for these domains are scarce and expensive to annotate, so few-shot classification is hard.
Approach
Combine an unsupervised masked-diffusion pretraining stage (shared encoder with modality-specific decoders, reconstructing visible and masked patches) with a CLIP-style contrastive fine-tuning stage that aligns the image encoder with class-description text embeddings from a transformer text encoder trained from scratch.
Key results
- On Houston with 2-shot, DiffCLIP reaches 52.15% OA versus 35.83% for CLIP (a 16.32-point gain).
- Across three remote sensing datasets, the paper reports an average 10.65% OA improvement over CLIP with 2-shot image-text pairs.
- On MRNet with 10 training samples, DiffCLIP reports ACC 0.763 and AUC 0.787, above SSL-DcGaR (0.731 / 0.758).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace baseline CLIP with DiffCLIP | CLIP · Houston, 2-shot | OA: 35.83 -> 52.15 | Helped |
| Replace baseline CLIP with DiffCLIP | CLIP · MUUFL, 2-shot | OA: 51.62 -> 59.39 | Helped |
| Replace baseline CLIP with DiffCLIP | CLIP · Trento, 2-shot | OA: 85.33 -> 93.19 | Helped |
| Add more unsupervised mask-diffusion samples (ablation) | DiffCLIP · Houston, unsupervised sample-count ablation | OA: 91.75 -> 92.58 -> 94.80 | Helped |
| Add the text encoder (ablation) | DiffCLIP · Houston | OA: 89.35 -> 94.80 | Helped |
| Add the diffusion process (ablation) | DiffCLIP · Houston | OA: 89.67 -> 94.80 | Helped |
| Add the masking process (ablation) | DiffCLIP · Houston | OA: 90.78 -> 94.80 | Helped |
| Add unsupervised mask diffusion pretraining (ablation) | DiffCLIP · Houston | OA: 87.82 -> 94.80 | Helped |
| Use prompt p2 instead of p1 (ablation) | DiffCLIP · Houston2013 | OA: 97.61 -> 97.05 | Hurt |
| Use prompt p3 instead of p1 (ablation) | DiffCLIP · Houston2013 | OA: 97.61 -> 97.63 | Helped |
| Use prompt p4 instead of p1 (ablation) | DiffCLIP · Houston2013 | OA: 97.61 -> 97.93 | Helped |
| Use specific class descriptions (p5) instead of p1 (ablation) | DiffCLIP · Houston2013 | OA: 97.61 -> 98.15 | Helped |
Surprising findings
- The text encoder is trained from scratch without loading pre-trained text weights, which the authors state is intentional to show the method's effectiveness.
Limitations
- Comparisons with 10 supervised and few-shot baselines use differing sample settings, and the 2-shot results are low in absolute terms (about 52% OA on Houston).
- The text encoder is not pretrained, and class descriptions are hand-written per dataset.
- Evaluated only on a few small hyperspectral/LiDAR benchmarks, with no broader natural-image or large-scale validation.
Experimental setup
Datasets: Houston Trento MUUFL MRNet
Benchmarks: Houston MUUFL Trento MRNet
Metrics: overall accuracy average accuracy Kappa coefficient accuracy AUC sensitivity specificity
Baselines: CLIP LDS2AE MFRN-ML SCFormer HIPL MMPR GLT CALC MIViT RN-FSC
Builds on: CLIP masked autoencoders denoising diffusion models ViT
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.88)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.87)
- SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing (AAAI 2024 · similarity 0.87)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.86)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.