Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Learning from True-False Labels via Multi-modal Prompt Retrieving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes True-False Labels (TFLs), a binary per-instance annotation of whether an image matches a randomly sampled candidate label, plus a risk-consistent estimator and a convolutional multi-modal prompt retrieving method (TMP) built on frozen CLIP ViT-L/14.
Problem
Zero-shot VLM labels are often wrong, and existing weakly supervised methods (SSL, PLL, CLL) fail to exploit VLM knowledge for accurate labels; the paper asks how to get more accurate and cheaper weak labels for multi-class classification.
Approach
Replace the multi-class label with a binary true/false answer about a randomly sampled candidate label, derive a risk-consistent estimator using conditional probabilities estimated by CLIP, and retrieve textual and visual prompt embeddings via a small CNN over a constant matrix to adapt CLIP.
Key results
- TMP (VLMs) averages 81.71% accuracy across five datasets, versus 79.80% for zero-shot CLIP and 81.14% for fully supervised CLIP LP on VLM labels.
- TMP (human) reaches 82.07% average and 93.55% on Food-101, comparable to fully supervised CLIP LP (94.94%).
- Weakly supervised baselines (ORCA, NACH, PaPi, CLL with WL) average 20-49% under VLM-based TFLs, more than 10 points below TMP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TMP with VLM-generated TFLs instead of zero-shot CLIP | CLIP ViT-L/14 · CIFAR-100, Tiny ImageNet, Caltech-101, Food-101, Stanford Cars | accuracy | Helped |
| Use TMP with human-generated TFLs instead of TMP with VLM-generated TFLs | CLIP ViT-L/14 · CIFAR-100, Tiny ImageNet, Caltech-101, Food-101, Stanford Cars | accuracy: 81.71 -> 82.07 | No clear effect |
| Add MPR (ablation) | TMP (human) · Caltech-101 | accuracy: 88.81 -> 90.60 | Helped |
| Add MPR (ablation) | TMP (human) · Food-101 | accuracy: 93.47 -> 93.55 | Helped |
| Add MPR (ablation) | TMP (human) · Stanford Cars | accuracy: 71.25 -> 72.60 | Helped |
| Add TPR (ablation) | TMP (human) · Caltech-101 | accuracy: 89.80 -> 90.60 | Helped |
| Add TPR (ablation) | TMP (human) · Food-101 | accuracy: 93.53 -> 93.55 | Helped |
| Add TPR (ablation) | TMP (human) · Stanford Cars | accuracy: 72.25 -> 72.60 | Helped |
| Add VPR (ablation) | TMP (human) · Caltech-101 | accuracy: 90.23 -> 90.60 | Helped |
| Add VPR (ablation) | TMP (human) · Food-101 | accuracy: 93.52 -> 93.55 | Helped |
| Add VPR (ablation) | TMP (human) · Stanford Cars | accuracy: 72.53 -> 72.60 | Helped |
| Increase the conditional probability hyperparameter λ from 0 to 0.1 (ablation) | TMP (human) · Caltech-101 | accuracy | Helped |
Surprising findings
- CLIP linear probe trained only on TFL instances with s=1 (CLIP LPT) collapses to about 27.55% average accuracy, far below zero-shot CLIP.
- Training on CLIP-generated TFLs improved the CLL baseline on CIFAR-100 by over 4 points relative to human TFLs, which the authors attribute to correcting dataset noise.
Limitations
- Does not use test data to improve performance; no test-time prompt adaptation.
- Labeling speed claims rest on a unit-time model of annotator effort rather than measured human studies.
- Evaluated only on five image classification datasets with CLIP ViT-L/14 as the sole backbone.
Experimental setup
Datasets: CIFAR-100 Tiny ImageNet Caltech-101 Food-101 Stanford Cars
Benchmarks: CIFAR-100 Tiny ImageNet Caltech-101 Food-101 Stanford Cars
Metrics: classification accuracy TFL annotation accuracy labeling speed multiplier training time per epoch
Baselines: ORCA NACH PaPi CLL with WL CLIP linear probe Zero-shot CLIP CLIP LPT
Builds on: CLIP CoOp MaPLe complementary-label learning unbiased risk estimator CoCoOp
Closest papers by meaning
- Losses over Labels: Weakly Supervised Learning via Direct Loss Construction (AAAI 2023 · similarity 0.88)
- Learning with Complementary Labels Revisited: The Selected-Completely-at-Random Setting Is More Practical (ICML 2024 · similarity 0.87)
- Complementary Label Learning with Positive Label Guessing and Negative Label Enhancement (ICLR 2025 · similarity 0.87)
- Learning with Partial-Label and Unlabeled Data: A Uniform Treatment for Supervision Redundancy and Insufficiency (ICML 2024 · similarity 0.87)
- Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions (CVPR 2025 · similarity 0.86)
- Realistic Evaluation of Deep Partial-Label Learning Algorithms (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.