Research map / Label-efficient and robust learning / Self-supervised and contrastive
seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes seq-JEPA, a joint-embedding predictive world model that processes sequences of action-conditioned views with a transformer aggregator, yielding separate equivariant encoder features and invariant aggregate features without equivariance losses.
Problem
The invariance-equivariance trade-off in joint-embedding self-supervised learning: two-view methods enforce either invariance (good for classification) or equivariance (good for fine-grained transformation-related tasks), but not both.
Approach
Process short sequences of (view, relative-action) pairs through a transformer aggregator with an [AGG] token, and train an action-conditioned predictor to predict the next view's representation. Per-view encoder features become equivariant to actions while the aggregate representation becomes invariant, without explicit equivariance losses or dual predictors.
Key results
- On 3DIEBench, seq-JEPA (M_tr=3, M_val=5) reaches 87.41% top-1 classification with rotation R^2 0.79-level equivariance, matching the best equivariant baselines' rotation R^2 while exceeding them in classification.
- On STL-10 predictive learning across saccades, seq-JEPA reaches 83.44% top-1 linear probe accuracy, comparable to SimCLR's 85.23% trained with full-resolution images and strong augmentations.
- Removing action conditioning drops rotation R^2 from 0.71 to 0.29 on 3DIEBench while classification remains high.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add action conditioning to the sequence model (ablation) | seq-JEPA on 3DIEBench · 3DIEBench | rotation prediction R²: 0.29 -> 0.71 | Helped |
| condition only the predictor rather than removing action conditioning (ablation) | seq-JEPA · 3DIEBench | rotation prediction R² | Helped |
| remove saliency-based fixation sampling (ablation) | seq-JEPA across saccades · STL-10 | classification (top-1): 83.44 -> 79.85 | Hurt |
| remove inhibition of return (ablation) | seq-JEPA across saccades · STL-10 | classification (top-1): 83.44 -> 77.97 | Hurt |
| use random uniform patch sampling instead of saliency-based sampling (ablation) | seq-JEPA across saccades · STL-10 | classification (top-1) | Hurt |
| increase inference sequence length from M_val=4 to M_val=6 | seq-JEPA across saccades · STL-10 | classification (top-1) | Helped |
| replace sequence aggregation with pairwise prediction losses (ablation) | Conv-JEPA versus seq-JEPA · STL-10 | classification (top-1) | Hurt |
| increase training sequence length | seq-JEPA · CIFAR100 and Tiny ImageNet | classification accuracy | Hurt |
| increase training sequence length | seq-JEPA · 3DIEBench and STL-10 | classification accuracy | Helped |
| increase inference sequence length | seq-JEPA · all datasets | classification accuracy | Helped |
| replace MLP projectors with transformer projectors (ablation) | SSL baselines · 3DIEBench | classification and equivariance performance | No clear effect |
| condition on both rotation and color | seq-JEPA · 3DIEBench | classification (top-1): 79.31 | Hurt |
Surprising findings
- Ablating action conditioning sharply reduces equivariance yet keeps classification accuracy high, indicating the aggregate representation is invariant regardless of action input.
- All methods, including seq-JEPA, fail at out-of-distribution rotation decoding with R^2 near -1 on the unseen rotation range.
Limitations
- Relies on access to known transformation group and transformation parameters (e.g., SO(3)).
- Only relatively short training and inference sequence lengths were explored due to end-to-end backbone training.
- Equivariance fails to generalize to out-of-distribution rotation ranges.
Experimental setup
Datasets: 3DIEBench CIFAR100 Tiny ImageNet STL-10 ImageNet-1k
Benchmarks: 3DIEBench CIFAR100 Tiny ImageNet STL-10 ImageNet-1k
Metrics: top-1 linear probe accuracy R^2 action decoding MRR Hit@1 Hit@5 path integration R^2
Baselines: SimCLR BYOL VICReg VICRegTraj SEN EquiMod SIE ContextSSL Conditional BYOL Conv-JEPA
Builds on: I-JEPA BYOL EquiMod SIE ViT
Closest papers by meaning
- Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning (NeurIPS 2024 · similarity 0.88)
- How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · similarity 0.87)
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels (NeurIPS 2026 · similarity 0.87)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.87)
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures (NeurIPS 2026 · similarity 0.86)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.