AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning

Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, Chuan Guo

NeurIPS 2023 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase VICReg training epochs from 250 to 1000 (ablation)VICReg · ImageNet 300K train setdeja vu score: from under 10% to over 30%Hurt
increase VICReg training epochs beyond 250 (ablation)VICReg · ImageNet 300K train setlinear probe train-test gapNo clear effect
increase training set size from 100K to 500K (ablation)VICReg · ImageNet, 1000 epochsdeja vu scoreNo clear effect
increase training set size (ablation)VICReg · ImageNet, 1000 epochslinear probe train-test gapImproved
increase VICReg invariance coefficient lambda above 25 (ablation)VICReg · ImageNet 100K train setdeja vu scoreImproved
vary SimCLR temperature (ablation)SimCLR · ImageNetdeja vu scoreMixed
switch SSL criterion from VICReg or Barlow Twins to SimCLR or BYOL (ablation)SSL training criterion · ImageNetdeja vu scoreImproved
use SSL instead of supervised training (ablation)ResNet101 on ImageNet · ImageNetdeja vu scoreHurt
use larger model (ResNet50/101 vs ResNet18/34; larger ViT) (ablation)VICReg · ImageNetdeja vu scoreHurt
use backbone embedding instead of projector embedding (guillotine regularization) (ablation)VICReg · ImageNetdeja vu scoreImproved
fine-tune pretrained VICReg for the first 5 epochs (ablation)VICReg pretrained 1000 epochs · ImageNet set A classification fine-tuningdeja vu scoreImproved
continue fine-tuning to 20 epochs (ablation)VICReg pretrained 1000 epochs · ImageNet set A classification fine-tuningdeja vu score: 5 epochs, 20 epochsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy