Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning predictable and robust neural representations by straightening image sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-supervised objective that explicitly straightens temporal trajectories of image-sequence embeddings, yielding predictive, factorized, and more noise- and adversarially-robust representations than invariance-based SSL.
Problem
Whether representational straightening (linear predictability of temporal trajectories), inspired by primate vision, can serve as a primary self-supervised learning objective, and how it compares to invariance-based SSL in predictability and robustness.
Approach
Directly optimize the average cosine similarity between successive difference vectors of embeddings over temporally smooth synthetic sequences, with variance and covariance whitening terms to prevent collapse; the straightened space permits linear extrapolation as prediction, and the loss can be added as a regularizer to existing SSL methods.
Key results
- Straightness loss yields progressively straighter embeddings across layers, while invariance objective ends slightly more curved than pixel input on sequential MNIST.
- Straightened representations are substantially more robust to Gaussian noise and white-box PGD L2 attacks than invariance-trained ResNet-18 on sequential CIFAR-10, with negligible clean accuracy loss.
- Adding the straightening loss as a regularizer improved adversarial robustness of Barlow Twins, SimCLR, W-MSE and DINO across attack budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use straightening instead of augmentation invariance | ResNet-18 on sequential CIFAR-10 · CIFAR-10 with Gaussian noise | classification accuracy | Helped |
| Use straightening instead of augmentation invariance | ResNet-18 on sequential CIFAR-10 · CIFAR-10 adversarial attacks | classification accuracy | Helped |
| Use straightening instead of augmentation invariance | ResNet-18 on sequential CIFAR-10 · CIFAR-C, highest-intensity corruptions | classification accuracy | Mixed |
| Use straightening instead of augmentation invariance | ResNet-18 on EgoGesture · EgoGesture video | classification accuracy under Gaussian noise | Helped |
| Add straightening regularization to the existing SSL objective | Barlow Twins, SimCLR, W-MSE, and DINO · CIFAR-10 adversarial attacks | classification accuracy | Helped |
| Add straightening regularization to the existing SSL objective | Barlow Twins, SimCLR, W-MSE, and DINO · CIFAR-10 pixel-level white noise | classification accuracy | Helped |
| Train the straightening objective on temporally shuffled frames (ablation) | Straightening model · Sequential MNIST | visual-feature decoding performance | Hurt |
| Apply straightening at multiple network stages instead of only the last layer (ablation) | Straightening model · CIFAR-10 | robustness | Helped |
| Replace straightening with temporal invariance | Self-supervised representations · Sequential MNIST dynamic-feature decoding | decoding accuracy | Hurt |
| Use linear extrapolation of straightened representations rather than the previous-frame representation | Straightening model · Sequential MNIST next-frame feature prediction | prediction performance | Helped |
Surprising findings
- Training the straightening objective on temporally shuffled frames gives poor decoding performance, showing temporal order is critical.
- Straightened representations fail to predict abrupt motion discontinuities such as direction reversals and boundary bounces.
Limitations
- Straightening is applied to synthetic augmented sequences rather than natural videos, and the learned model captures smooth transformations but not macro-structure of transformation statistics.
- Feedforward single-timescale prediction may be insufficient to exploit hierarchical temporal structure in natural videos.
- Gains on natural video (EgoGesture) were shown only with depth input and no augmentations, on a small setup.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-C EgoGesture ImageNet VID Objectron
Benchmarks: sequential MNIST decoding CIFAR-10 linear classification CIFAR-C EgoGesture
Metrics: straightness (cosine similarity) classification accuracy decoding accuracy adversarial robustness accuracy vs attack budget participation ratio
Baselines: Barlow Twins SimCLR W-MSE DINO invariance objective with whitening
Builds on: Barlow Twins solo-learn projected gradient descent
Closest papers by meaning
- Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening (NeurIPS 2025 · similarity 0.85)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.85)
- Towards flexible perception with visual memory (ICML 2025 · similarity 0.84)
- AI-Generated Video Detection via Perceptual Straightening (NeurIPS 2025 · similarity 0.84)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.84)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.