Research map / Label-efficient and robust learning / Self-supervised and contrastive

No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Invariance-based self-supervised learning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FUNGI, which concatenates randomly projected self-supervised gradients (KL, DINO, SimCLR) from a frozen transformer's last attention projection with its embedding to improve kNN classification, retrieval, and in-context segmentation without training.

Problem

Frozen pretrained transformer embeddings are used directly for kNN classification and retrieval, but they may miss task-specific information; the paper asks whether self-supervised gradients can enrich these features without any training.

Approach

Compute per-sample gradients of self-supervised losses (KL-to-uniform, DINO, SimCLR) with respect to a late linear layer of a frozen backbone using a randomly initialized projection head, reduce them with binary random projections, L2-normalize, and concatenate with the embedding before PCA to form retrieval features.

Key results

• On 11 image datasets with AugReg IN1K ViT-B/16, adding KL+DINO+SimCLR gradients raises mean kNN accuracy from 59.2% to 63.6% (full data) and DINO ViT-B/16 semantic segmentation in-context retrieval on VOC gains up to 17.2 mIoU points at smallest memory bank.
• FUNGI with DINO ViT-B/16 reaches within 3.5% mIoU of HummingBird on Pascal VOC without any training.
• On the text modality, KL+SimCLR gradients improve BERT full-data kNN on TREC from 83.6 to 86.8, while SimCLR gradients hurt some datasets.

Measured effects reported in the paper

Surprising findings

• Gains diminish as pretraining data grows: smaller relative improvements for IN21K-pretrained backbones and EVA-CLIP versus CLIP.
• For linear probing, FUNGI often hurts self-supervised backbones such as DINO and DINOv2, while helping supervised ones.

Limitations

• SimCLR gradients are inconsistent and sometimes degrade performance, especially for text and audio.
• Benefits diminish for backbones pretrained on larger datasets (IN21K, EVA-CLIP) and for self-supervised backbones in linear probing.
• Extraction adds computational overhead from multiple self-supervised forward/backward passes per image.

Experimental setup

Datasets: Cars CUB DTD EuroSAT CIFAR-100 CIFAR-10 Oxford Pets Food-101 ImageNet-1K FGVC Aircraft Flowers102 TREC

Benchmarks: Cars CUB DTD EuroSAT CIFAR-100 CIFAR-10 Oxford Pets Food-101 ImageNet-1K FGVC Aircraft Flowers102 TREC

Metrics: top-1 accuracy mIoU mAP mean accuracy

Baselines: DINO embeddings HummingBird end-to-end fine-tuning CLIP embeddings EVA-CLIP embeddings

Builds on: DINO SimCLR Head2Toe retrieval-based in-context scene understanding (Balazevic et al.) binary random projections

Closest papers by meaning

• Towards Understanding Self-Pretraining for Sequence Classification (NeurIPS 2026 · similarity 0.85)
• Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.85)
• Vision Transformers Need Registers (ICLR 2024 · similarity 0.84)
• Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding (CVPR 2025 · similarity 0.84)
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.84)
• Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.84)