Research map / Label-efficient and robust learning / Self-supervised and contrastive

Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features

AAAI 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies discriminative features in self-supervised ResNet representations, builds an unsupervised sample-wise Q-Score that predicts linear-probe misclassifications, and uses it as a fine-tuning regularizer to improve linear accuracy.

Problem

Self-supervised representations are noisy and hard to interpret, and their failure modes (sample-level misclassification in linear evaluation) are poorly understood, without access to labels.

Approach

Selects moderately activating ("discriminative") feature coordinates by their activation frequency across the population without labels, defines a per-sample Q-Score over highly activating discriminative features that predicts correct vs. incorrect classification, and uses it as a regularizer during further pre-training with an L1-norm penalty to avoid trivial solutions.

Key results

• Q-Score regularization improves linear probing accuracy for all eight SSL baselines, with BYOL gaining 5.8% on ImageNet-100 and 3.7% on ImageNet-1K.
• Q-Score predicts misclassified samples with AUPRC up to 91.45 on ImageNet-100 and 78.78 on ImageNet-1K (BYOL).
• Discriminative features allow compressing the representation by up to 40% with little loss in linear classification accuracy.

Measured effects reported in the paper

Surprising findings

• Lasso (L1) regularization degrades linear probing accuracy for most models, showing that naive sparsification loses information.
• Q-Score's AUROC/AUPRC on supervised (non-SSL) ResNets is lower than on SSL representations, suggesting the property is more prominent in self-supervised learning.

Limitations

• Analysis is restricted to ResNet-based encoders; ViT-based SSL representations are not studied and may behave differently.
• Hyperparameters (lambda1, lambda2, epsilon) were not extensively tuned due to limited resources.
• Findings show only association between discriminative features and classification accuracy, not causation.

Experimental setup

Datasets: ImageNet-1K ImageNet-100 CIFAR-10 CIFAR-100 STL-10 Salient ImageNet Aircraft Food Cars DTD Flowers

Benchmarks: ImageNet-100 ImageNet-1K CIFAR-10 CIFAR-100 STL-10 Salient ImageNet

Metrics: top-1 linear probing accuracy AUPRC AUROC mIoU

Baselines: Lasso (L1) regularization SimCLR SwaV MoCo BYOL DINO SimSiam VICReg Barlow Twins

Builds on: SimCLR solo-learn Salient ImageNet GradCAM LARS

Closest papers by meaning

• Improving Self-Supervised Learning by Characterizing Idealized Representations (NeurIPS 2022 · similarity 0.88)
• $\alpha$-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay (NeurIPS 2022 · similarity 0.88)
• Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.87)
• Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.86)
• Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.85)
• Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.85)