Research map / Video / Video segmentation and tracking
Learning from One Continuous Video Stream
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a framework and benchmark streams (Ego4D-stream, ScanNet-stream) for online learning from one continuous video stream with batch size 1, using pixel-to-pixel future prediction pretraining and an optimizer/update-frequency study, matching IID batch-size-1 learning out-of-stream.
Problem
Online learning from a single highly correlated continuous video stream (batch size 1, no shuffling) is largely unexplored; standard deep learning tools fail there, and adaptation and generalization need to be measured jointly.
Approach
Frame all tasks as pixel-to-pixel RGB prediction with a single L2 loss so pretraining and single-stream learning share architecture and loss; evaluate both in-stream adaptation and out-of-stream generalization; find that momentum hurts, infrequent updates help generalization, and video future-prediction pretraining beats ImageNet pretraining.
Key results
- Baby Learning (continuous, batch size 1) matches IID batch-size-1 STDL out-of-stream on ScanNet segmentation at displacement 1 (0.011 vs 0.010 in-stream/out-of-stream range reported in Table 4) while outperforming it in-stream (0.463 vs 0.376 in-stream mIoU at t+1 is not better, so gains are mixed;
- Future-prediction pretraining (Guided) reduces out-of-stream Ego4D L2 from 0.044 (ImageNet-MAE) to 0.043 and ScanNet depth out-of-stream from 2.045 to 1.990 (Table 3).
- Without momentum, RMSprop-type optimizers substantially outperform Adam variants in single-stream learning (Figure 4, averaged over 8 settings).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use momentum-free optimizer (RMSprop) instead of AdamW (ablation) | single-stream learning · Ego4D-stream and ScanNet-stream segmentation | mean IoU / L2 (averaged sweep) | Helped |
| Lower momentum of AdamW (ablation) | AdamW optimizer in single-stream learning · ScanNet-stream segmentation | mean IoU | Helped |
| Less frequent weight updates (16 steps, 0.64s) vs every step (ablation) | ViT and UNet single-stream learning · Ego4D-stream and ScanNet-stream | out-of-stream generalization and in-stream adaptation | Mixed |
| Cosine decay learning rate schedule instead of constant learning rate (ablation) | single-stream learning · ScanNet-stream segmentation | in-stream and out-of-stream mean IoU | Mixed |
| Add replay buffer (10,000 samples, batch size 4) (ablation) | single-stream learning · ScanNet-stream segmentation (displacement 1) | mean IoU: +2% mean IoU | No clear effect |
| Pretrain ViT-L with guided future prediction instead of ImageNet-1K MAE (ablation) | ViT-L single-stream learning · ScanNet-stream segmentation (out-of-stream) and Ego4D-stream | mean IoU / L2 | Mixed |
| Pretrain on future prediction vs no pretraining (ablation) | ViT-L single-stream learning · Ego4D-stream and ScanNet-stream | average L2 / IoU / log RMSE | Helped |
| Use BL (continuous bs 1) instead of STDL IID bs 1 on ScanNet segmentation | ViT-L · ScanNet-stream segmentation | in-stream mean IoU | Helped |
| Continuous stream BL vs STDL IID bs 1 on out-of-stream metrics | ViT-L · Ego4D-stream, ScanNet depth, ScanNet segmentation | out-of-stream L2 / log RMSE / IoU | No clear effect |
| Longer temporal displacement for future prediction pretraining (ablation) | ViT-L pretraining on Kinetics · Kinetics linear top-1 accuracy | top-1 accuracy | Helped |
Surprising findings
- Momentum, default in Adam, hurts single-stream learning; lowering AdamW momentum recovers part of RMSprop's performance.
- Longer temporal displacement in future-prediction pretraining gives better Kinetics linear accuracy, even for masked prediction that reduces to MAE at displacement 0.
Limitations
- Gains over IID batch-size-1 learning are mixed; BL does not clearly beat STDL in all metrics and continuous streams still lag IID setups.
- Explicit memory modules and long-context models were not explored.
- Streams are concatenations of existing datasets rather than truly long natural streams (24h experiments).
Experimental setup
Datasets: Ego4D ScanNetV2 Kinetics-700-2020 ImageNet
Benchmarks: Ego4D-stream future prediction ScanNet-stream semantic segmentation ScanNet-stream depth prediction ScanNet-stream pixel prediction Kinetics linear probe
Metrics: average L2 pixelwise distance mean per-frame IoU recall logRMSE linear top-1 accuracy
Baselines: STDL (IID) bs 16 STDL (IID) bs 1 Blind (dummy) model ImageNet-MAE pretraining ImageNet classification pretraining Adam/AdamW
Builds on: video MAE Siamese MAE replay buffers ImageNet MAE
Closest papers by meaning
- Summarizing Stream Data for Memory-Constrained Online Continual Learning (AAAI 2024 · similarity 0.87)
- Learning from Streaming Video with Orthogonal Gradients (CVPR 2025 · similarity 0.86)
- Learning from Snapshots of Discrete and Continuous Data Streams (NeurIPS 2024 · similarity 0.86)
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.84)
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding (NeurIPS 2025 · similarity 0.84)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.