Research map / Label-efficient and robust learning / Self-supervised and contrastive
Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C-JEPA, which adds VICReg variance/covariance and invariance regularization to I-JEPA to prevent collapse and improve convergence of image representation learning, evaluated on ImageNet linear probing/fine-tuning and dense/video/low-level downstream tasks.
Problem
I-JEPA's EMA-based target encoder does not reliably prevent full representation collapse, and its prediction objective does not accurately learn the mean of patch representations, slowing and limiting representation learning.
Approach
Integrates VICReg's variance, covariance and invariance regularization terms into the I-JEPA masked patch-prediction objective, with a theoretical argument (linear predictor eigenspace analysis, SimSiam connection) that the predictor and stop-gradient alone yield vanishing or collapsing dynamics.
Key results
- On ViT-B/16 with 600 pretraining epochs, C-JEPA reaches 73.7 linprob and 84.5 fine-tune on ImageNet-1K, versus 72.9 and 83.5 for I-JEPA.
- On ViT-L/16, C-JEPA improves DAVIS (J&F)m from 56.6 to 58.3 and Clevr/Count from 85.6 to 86.8 over I-JEPA.
- Ablations on ViT-T/16 at 100 epochs show the full VICReg setup gives 69.5 linprob versus 63.7 for I-JEPA alone in the convergence ablation table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VICReg regularization to I-JEPA | I-JEPA with ViT-B/16 · ImageNet-1K linear probing | linprob: 72.9 -> 73.7 | Helped |
| Add VICReg regularization to I-JEPA | I-JEPA with ViT-B/16 · ImageNet-1K fine-tuning | fine-tune accuracy: 83.5 -> 84.5 | Helped |
| Add VICReg regularization to I-JEPA | I-JEPA with ViT-B/16 · MS-COCO detection and instance segmentation | APbox and APmask: 49.9 -> 50.7; 44.5 -> 45.3 | Helped |
| Add VICReg regularization to I-JEPA | I-JEPA with ViT-B/16 · ADE20K semantic segmentation | mIoU: 47.6 -> 48.7 | Helped |
| Add VICReg regularization to I-JEPA | I-JEPA with ViT-L/16 · DAVIS-2017 video object segmentation | (J & F)m: 56.6 -> 58.3 | Helped |
| Add Variance/Covariance terms (ablation) | I-JEPA with ViT-B/16, 100 epochs · ImageNet-1K linear probing | linprob: 63.7 -> 68.3 | Helped |
| Add Invariance term to Variance/Covariance-regularized I-JEPA (ablation) | I-JEPA with ViT-B/16, 100 epochs · ImageNet-1K linear probing | linprob: 68.3 -> 69.5 | Helped |
| Replace Variance/Covariance regularization with EMA for collapse and add Invariance for mean (ablation) | I-JEPA with ViT-B/16, 100 epochs · ImageNet-1K linear probing | linprob: 69.5 -> 67.6 | Hurt |
| Add Variance/Covariance terms (ablation) | I-JEPA with ViT-B/16, 600 epochs · ImageNet-1K linear probing | linprob: 72.9 -> 73.5 | Helped |
| Add Invariance term to Variance/Covariance-regularized I-JEPA (ablation) | I-JEPA with ViT-B/16, 600 epochs · ImageNet-1K linear probing | linprob: 73.5 -> 73.7 | Helped |
| Vary VICReg coefficient β_vicreg from 0.001 to 0.01 (ablation) | C-JEPA with ViT-T/16 · DAVIS-2017 video object segmentation | (J & F)m: 50.2 -> 31.9 | Hurt |
| Vary VICReg coefficient β_vicreg from 0.001 to 0.1 (ablation) | C-JEPA with ViT-T/16 · DAVIS-2017 video object segmentation | (J & F)m: 50.2 -> Collapsing | Hurt |
Surprising findings
- Large VICReg coefficients (beta_vicreg of 0.1) cause collapse in the DAVIS sensitivity study, and performance drops sharply above 0.005.
Limitations
- Gains over I-JEPA are modest (about 0.8 to 1.2 points on most benchmarks) and single-seed results are reported without variance.
- Theoretical analysis relies on linear predictors, NTK approximations and gradient-flow assumptions that may not hold for deep ViTs.
- Performance is sensitive to the VICReg coefficient, which collapses at high values.
Experimental setup
Datasets: ImageNet-1K MS-COCO ADE20K DAVIS-2017 CLEVR
Benchmarks: ImageNet-1K MS-COCO ADE20K DAVIS-2017 CLEVR
Metrics: top-1 accuracy (linear probing) top-1 accuracy (fine-tune) AP box AP mask mIoU J&F mean Clevr/Count Clevr/Dist
Baselines: I-JEPA DINO BEiT MAE iBOT data2vec
Builds on: I-JEPA VICReg SimSiam
Closest papers by meaning
- Denoising with a Joint-Embedding Predictive Architecture (ICLR 2025 · similarity 0.89)
- How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · similarity 0.89)
- seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models (NeurIPS 2025 · similarity 0.88)
- You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning (NeurIPS 2024 · similarity 0.87)
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures (NeurIPS 2026 · similarity 0.86)
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.