Research map / Label-efficient and robust learning / Recognition and long-tail
Multi-Label Supervised Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-level contrastive learning (MLCL), which attaches multiple projection heads (one per label or hierarchy level plus a global head) to an encoder, each with its own temperature, to capture multiple similarity notions in supervised contrastive learning.
Problem
Supervised contrastive learning relies on a single projection space and a single label per sample, which limits its use for multi-label and hierarchical classification, especially with limited training data.
Approach
Use multiple projection heads, each defining positives by a different similarity criterion (a single label level, or Jaccard label overlap for a global head), with head-specific temperatures (low for fine labels, high for superclasses) and a weighted sum of the per-head SupCon losses.
Key results
- On full CIFAR-100, MLCL reaches 77.70% top-1 accuracy versus 76.50% for SupCon, with 250 epochs versus 1,000.
- With 1,000 CIFAR-100 training samples, MLCL reaches 34.74% versus 26.82% for SupCon.
- On TripAdvisor with 30 training samples, MLCL reaches 72.9% versus 71.30% for cross-entropy BERT fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MLCL instead of SupCon | CIFAR-100 image classification · CIFAR-100 test | Top-1 accuracy: 76.50 -> 77.70 | Helped |
| Use MLCL instead of HiMulConE | DeepFashion image classification · DeepFashion test | Top-1 accuracy: 73.21 -> 73.90 | Helped |
| Use MLCL instead of cross-entropy | BERT multi-label classifier · TripAdvisor and BeerAdvocate, 30–360 samples | accuracy: TripAdvisor: 71.30 -> 72.9; 74.19 -> 75.1; 77.00 -> 77.9; 78.10 -> 79.0; BeerAdvocate: 65.45 -> 66.90; 66.37 -> 67.40; 6 | Helped |
| Add the global projection head (ablation) | MLCL on TripAdvisor · TripAdvisor, 30–360 samples | accuracy: 72.50 -> 72.9; 74.81 -> 75.1; 77.30 -> 77.9; 78.44 -> 79.0 | Helped |
| Add the global projection head (ablation) | MLCL on BeerAdvocate · BeerAdvocate, 30–360 samples | accuracy: 65.95 -> 66.90; 66.71 -> 67.40; 69.25 -> 69.61; 71.22 -> 71.81 | Helped |
| Use MLCL rather than SupCon to train the transferred encoder | CIFAR-100-to-CIFAR-10 transfer · CIFAR-10 test | accuracy: 85.97 -> 86.88; 83.70 -> 84.91; 82.54 -> 83.69; 78.62 -> 79.44 | Helped |
| Use MLCL rather than SupCon with fewer training samples | CIFAR-100 image classification · CIFAR-100 test, 1,000–40,000 training samples | Top-1 accuracy: 19.31 -> 34.74; 34.51 -> 56.02; 40.20 -> 59.32; 54.73 -> 69.36; 59.26 -> 72.50; 63.21 -> 75.60 | Helped |
| Increase the superclass-head temperature to 0.5 (ablation) | MLCL hierarchical classification · CIFAR-100 test, subset of training samples | accuracy | Helped |
| Set the superclass-head temperature excessively high or low rather than 0.5 (ablation) | MLCL hierarchical classification · CIFAR-100 test, subset of training samples | accuracy | Hurt |
| Assign a large weight of 0.6 to global projection loss under uniform label noise (ablation) | MLCL multi-label classifier · TripAdvisor and BeerAdvocate noisy-label settings | Top-1 accuracy: up to 7% | Helped |
Surprising findings
- Higher temperature for the superclass head (0.5) works better than lower temperatures, since low temperatures focus on hard positives that are undesirable at higher hierarchy levels.
Limitations
- Improvement over SupCon on full CIFAR-100 is marginal (about 1%).
- Multi-label experiments use only two text datasets with sentiment labels; image experiments are hierarchical only.
- Hyperparameters such as head weights and temperatures are tuned per dataset.
Experimental setup
Datasets: CIFAR-100 DeepFashion TripAdvisor BeerAdvocate CIFAR-10
Benchmarks: CIFAR-100 DeepFashion TripAdvisor BeerAdvocate CIFAR-10
Metrics: top-1 classification accuracy
Baselines: SimCLR SupCon Guided HiMulConE Cross-Entropy
Builds on: SupCon SimCLR BERT fine-tuning
Closest papers by meaning
- Addressing Multi-Label Learning with Partial Labels: From Sample Selection to Label Selection (AAAI 2025 · similarity 0.88)
- Double-Layer Hybrid-Label Identification Feature Selection for Multi-View Multi-Label Learning (AAAI 2024 · similarity 0.87)
- In Pursuit of Causal Label Correlations for Multi-label Image Recognition (NeurIPS 2024 · similarity 0.86)
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning (CVPR 2025 · similarity 0.86)
- Weakly-Supervised Contrastive Learning for Imprecise Class Labels (ICML 2025 · similarity 0.86)
- CFDM: Contrastive Fusion and Disambiguation for Multi-View Partial-Label Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.