Research map / Video / Video segmentation and tracking
Learning from Streaming Video with Orthogonal Gradients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes orthogonal-gradient versions of SGD and AdamW so that self-supervised video models can learn from sequentially streamed (non-shuffled) video without collapsing due to correlated consecutive gradients.
Problem
Self-supervised video representation learning from a continuous stream of video breaks the IID batch assumption; consecutive batches have highly correlated gradients, causing performance drops or training failure.
Approach
Remove the component of the current gradient along the EMA of past clean gradients (projection-based orthogonal update) and feed this orthogonal gradient into standard optimizers such as SGD and AdamW, which reduces gradient correlation and recovers performance under sequential training.
Key results
- On DoRA with DINO ImageNet init on WTvenice, Orthogonal-AdamW gets 64.5 ImageNet linear probe top-1 vs 6.1 for AdamW (DINO baseline 74.4).
- VideoMAE on SSV2 with batch-along-time sequential loading: Orthogonal-AdamW attn-probe top-1 48.0 vs AdamW 46.1.
- Future prediction on Ego4D-Stream out-of-stream pixel MSE improves from 0.026 (AdamW) to 0.023 (Orthogonal-AdamW).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Orthogonal-AdamW instead of AdamW | DoRA sequential training initialized from DINOImageNet · ImageNet | linear probe top1: 6.1 -> 64.5 | Helped |
| use Orthogonal-AdamW instead of AdamW | DoRA sequential training initialized from VideoMAESSV2 · ImageNet | kNN top1: 3.0 to 5.7 | Helped |
| use Orthogonal-AdamW instead of AdamW | DoRA sequential training from random initialization · ImageNet | linear probe top1: 3.5 -> 8.2 | Helped |
| use sequential batch-along-time rather than shuffled clips | VideoMAE pretrained on SSV2 with AdamW · SSV2 | linear-probe top1: 19.0 -> 16.4 | Hurt |
| use Orthogonal-AdamW instead of AdamW | VideoMAE pretrained on SSV2, batch-along-time · SSV2 | attn-probe top1: 46.1 -> 48.0 | Helped |
| use Orthogonal-AdamW instead of AdamW | VideoMAE pretrained on SSV2, batch-along-video · SSV2 | linear-probe top1: 9.5 -> 10.4 | Helped |
| use Orthogonal-AdamW instead of AdamW | VideoMAE pretrained on K400, batch-along-time · SSV2 | linear-probe top1: 19.3 -> 20.5 | Helped |
| use Orthogonal-AdamW instead of AdamW | VideoMAE pretrained on K400, batch-along-video · SSV2 | attn-probe top1: 43.5 -> 43.6 | No clear effect |
| use Orthogonal-AdamW instead of AdamW | BabyLearning future prediction · Ego4D-Stream in-stream | Pixel MSE: 0.034 -> 0.031 | Helped |
| use Orthogonal-AdamW instead of AdamW | BabyLearning future prediction · Ego4D-Stream out-of-stream | Pixel MSE: 0.026 -> 0.023 | Helped |
| use Orthogonal-AdamW instead of AdamW | BabyLearning future prediction · ScanNet-Stream in-stream | Pixel MSE: 0.033 -> 0.032 | Helped |
| use Orthogonal-AdamW instead of AdamW | BabyLearning future prediction · ScanNet-Stream out-of-stream | Pixel MSE: 0.033 -> 0.033 | No clear effect |
Surprising findings
- Orthogonal-AdamW also slightly improves VideoMAE training on shuffled clips (e.g., linear probe +2% on SSV2).
- Sequential DoRA training with plain AdamW from DINO initialization damages the model so it cannot be trained further, while orthogonal AdamW prevents this failure.
Limitations
- Gains on several settings are small (1-2 points), and differences are from single runs without reported variance.
- DoRA from-scratch training is inefficient and only evaluated in a short schedule.
- Orthogonal update relies on a single-step/EMA gradient direction, which may be sensitive to noise, and the method is evaluated mainly with AdamW.
Experimental setup
Datasets: WalkingTour WTvenice Something-Something-V2 Kinetics-400 ScanNet-V2 Ego4D ImageNet
Benchmarks: ImageNet linear probe ImageNet kNN SSV2 linear probe SSV2 attentive probe Ego4D-Stream future prediction ScanNet-Stream future prediction
Metrics: top-1 accuracy pixel MSE PSNR
Baselines: AdamW SGD RMSProp DoRA VideoMAE BabyLearning
Builds on: DoRA VideoMAE Baby Learning AdamW DINO
Closest papers by meaning
- Learning from One Continuous Video Stream (CVPR 2024 · similarity 0.86)
- How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (ICLR 2022 · similarity 0.86)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.83)
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.83)
- Progressive Fourier Neural Representation for Sequential Video Compilation (ICLR 2024 · similarity 0.83)
- Boosting Neural Representations for Videos with a Conditional Decoder (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.