Research map / Vision-language and multimodal / Multimodal LLMs
Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies concept mismatch and concept confusion as causes of imbalanced VLM pseudolabels and proposes CAP, combining LLM-based concept alignment for initial pseudolabels with a confusion-aware calibrated margin loss and dual adapters, evaluated on six image classification benchmarks under UL, SSL, and TRZSL settings.
Problem
VLM zero-shot pseudolabels are imbalanced across classes, which hurts fine-tuning on unlabeled downstream data; the causes of this imbalance are underexplored.
Approach
Attributes pseudolabel imbalance to a semantic gap manifesting as concept mismatch (class text far from its visual cluster) and concept confusion (similar classes biased in zero-shot predictions). It detects mismatched classes via iterative clustering, enhances their text with LLM descriptions to select better initial pseudolabels, and uses a margin-based cross-entropy whose per-class margins are
Key results
- Under UL, CAP reaches 81.48% on RESISC45 and 76.80% on Flowers102 versus CPL's 77.39% and 72.90%, with 6.29% relative improvement over SoTA claimed.
- Ablation on RESISC45/DTD/Flowers102: both components together give 82.03/55.26/76.77 versus 68.41/49.57/70.85 without either.
- With ViT-L/14, CAP reaches 84.46% UL on the second dataset (EuroSAT-like row labeling ambiguous) and 65.85% UL on DTD versus CPL's 57.20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add concept alignment (CA) to the UL pipeline (ablation) | CAP on RESISC45 · RESISC45 UL ablation | accuracy: 68.41 -> 72.48 (+4.07) | Helped |
| Add confusion-aware calibrated margin (CACM) to the UL pipeline (ablation) | CAP on RESISC45 · RESISC45 UL ablation | accuracy: 68.41 -> 78.77 (+10.36) | Helped |
| Add concept alignment together with CACM (ablation) | CAP on RESISC45, DTD, Flowers102 · UL ablation (RESISC45, DTD, Flowers102) | accuracy: RESISC45 82.03 vs 78.77 CACM-only; DTD 55.26 vs 52.76; Flowers102 76.77 vs 74.49 | Helped |
| Add concept alignment (CA) alone on DTD (ablation) | CAP on DTD · DTD UL ablation | accuracy: 49.57 -> 53.13 (+3.56) | Helped |
| Add CACM alone on DTD (vs CA alone) (ablation) | CAP on DTD · DTD UL ablation | accuracy: 49.57 -> 52.76 (+3.19), slightly below CA-only 53.13 | Mixed |
| Full CAP vs CPL under UL | CAP vs CPL (ViT-B/32) · Flowers102, RESISC45, DTD, EuroSAT, CUB, FGVCAircraft | accuracy: Flowers102 76.80 vs 72.90; RESISC45 81.48 vs 77.39; DTD 55.29 vs 51.91; EuroSAT 75.01 vs 67.26; CUB 55.76 vs 53.47; FGVC | Helped |
| Full CAP vs CPL under SSL and TRZSL | CAP vs CPL (ViT-B/32) · CUB and FGVCAircraft, SSL and TRZSL | accuracy: CUB SSL 58.04 vs CPL 58.53; CUB TRZSL 61.35 vs 66.20; FGVC SSL 21.79 vs 22.48; FGVC TRZSL 29.03 vs 30.86 | Mixed |
| Use independent main and pseudo adapters vs shared adapters (ablation) | CAP adapter design · DTD, RESISC45, EuroSAT UL | accuracy: DTD 55.3 vs 54.6; RESISC45 81.5 vs 80.6; EuroSAT 76.2 vs 78.3 (shared better) | Mixed |
| Increase number of ViT-L/14 backbone (CAP with ViT-L/14 vs CPL) | CAP on DTD with ViT-L/14 · DTD UL | accuracy: 65.85 vs CPL 57.20 | Helped |
| Sensitivity to margin scale m (ablation) | CACM · RESISC45 and DTD UL | accuracy: accuracy relatively stable across m; m=12 default | No clear effect |
| Sensitivity to top-k (k from 12 to 20) (ablation) | CAP SSL · RESISC45 and DTD SSL | accuracy: RESISC45 82.9-83.5; DTD 60.6-61.5 | No clear effect |
| Sensitivity to confidence threshold tau (ablation) | CAP UL · RESISC45 and DTD UL | accuracy: RESISC45 80.5-81.6; DTD 55.9-57.1 | No clear effect |
Surprising findings
- Without independent adapters, the variant performs better than with them on EuroSAT (78.3 vs 76.2 under UL), contrary to the motivation for separate adapters.
- Concept alignment and CACM reported as complementary, with CACM alone giving larger gains than concept alignment alone on RESISC45 and Flowers102.
Limitations
- Relies on an external LLM (GPT-4o-mini) for text augmentation, adding cost and dependence on a closed API.
- Many hyperparameters (k, tau, m, t, n) are set manually; the margin scale m is fixed at 12.
- Evaluated only on image classification with CLIP backbones; the causal analysis of imbalance is supported by qualitative examples on few datasets.
Experimental setup
Datasets: Flowers102 RESISC45 DTD EuroSAT CUB FGVCAircraft
Benchmarks: Flowers102 RESISC45 DTD EuroSAT CUB FGVCAircraft
Metrics: test accuracy harmonic mean accuracy (TRZSL) local ECE
Baselines: zero-shot CLIP FPL GRIP CPL
Builds on: MaPLe CoOp FixMatch GRIP CPL UPL logit adjustment
Closest papers by meaning
- Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data (ICML 2024 · similarity 0.90)
- From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration (CVPR 2025 · similarity 0.87)
- Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.87)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.87)
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.86)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.