Research map / Video / Video segmentation and tracking
Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning strong image encoders from a single long, uncurated first-person "Walking Tours" video using a DoRA method (multi-object masking with Sinkhorn-Knopp-refined object prototypes and unsupervised tracking), rebuttal-stage responses only.
Problem
Self-supervised image representation learning typically needs billions of images; the paper asks whether strong image encoders can be learned from a single long unlabelled video instead.
Approach
Introduces the Walking Tours dataset of hours-long high-resolution first-person videos and the DoRA method, which uses object-level multi-object masking tracked across frames and Sinkhorn-Knopp refined object prototypes to learn dense-aware representations from video.
Key results
- DoRA (WT-all) fine-tuned outperforms DINO (ImNet) on all five rebuttal classification fine-tuning tasks (e.g., ImNet 81.4 vs 81.3), while linear probing on ImageNet remains lower (72 LP vs 45 LP reported for WT).
- On ADE20K and MS-COCO, DoRA with ViT-B/16 (WT-Venice) reaches 40.3 mIoU and 41.7 mAP, up 4.9 and 2.2 points over ViT-S/16.
- DINO pretrained on WT-Venice for 100 epochs reaches 54.6 on DAVIS versus 61.8 reported by DINO for ImNet at 300 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain DoRA on WT-all instead of WT-Venice | DoRA · ImageNet-1k k-NN | accuracy: +2% | Helped |
| Pretrain DoRA on WT-all instead of WT-Venice | DoRA · ADE20K semantic segmentation | mIoU: 1.5% | Helped |
| Pretrain DoRA on WT-all instead of WT-Venice | DoRA · ADE20K semantic segmentation | Acc_m: 2.5% | Helped |
| Pretrain DoRA on WT-all instead of WT-Venice | DoRA · MS-COCO instance segmentation | performance: 1.6% | Helped |
| Pretrain DoRA on WT-all instead of WT-Venice | DoRA · Object tracking | mAO: 4.5% | Helped |
| Increase encoder from ViT-S/16 to ViT-B/16 | DoRA (WT-Venice) · ADE20K | mIoU: 4.9% | Helped |
| Increase encoder from ViT-S/16 to ViT-B/16 | DoRA (WT-Venice) · MS-COCO | mAP: 2.2% | Helped |
| Use a smaller entropy coefficient and increase SK iterations to 60 (ablation) | DoRA · Downstream tasks | performance | Hurt |
| Add the iterative attended-region removal heuristic (ablation) | DoRA · ImageNet linear probing and Pascal VOC unsupervised object discovery | performance | Hurt |
| Add SK (ablation) | DoRA · Pretraining throughput | throughput: 1883 im/sec -> 1860 im/sec | Hurt |
| Increase WT-Venice pretraining from 100 to 300 epochs | DoRA · Image-based downstream tasks | performance | Helped |
| Use DoRA (WT-all) instead of DoRA (WT-Venice) | DoRA · Video object segmentation | performance | Hurt |
Surprising findings
- Training on a single walking-tour video (WT-Venice) achieves accuracy similar to training on all ten videos (WT-all) on several tasks.
Limitations
- Linear probing on ImageNet lags substantially behind ImageNet-pretrained DINO, attributed to domain gap.
- Experiments are mostly on ViT-S/16 and the WT-all setting does not clearly beat DINO on ImageNet-1k.
- Privacy and safety of the dataset (faces, legal issues) were not thoroughly assessed in the original submission.
Experimental setup
Datasets: Walking Tours ImageNet-1k Kinetics-400 Epic-Kitchens
Benchmarks: ImageNet linear probing ImageNet k-NN ADE20K MS-COCO DAVIS-2017 Places205 CIFAR-10 CIFAR-100 iNaturalist 2018 Oxford Flowers Stanford Cars Pascal VOC
Metrics: top-1 accuracy mIoU mAP mAO SR_0.5 SR_0.75 J&F
Baselines: DINO MAE VITO DINOv2 OpenCLIP
Builds on: DINO iBOT CutLER
Closest papers by meaning
- How Well Do Self-Supervised Models Transfer? (CVPR 2021 · similarity 0.88)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.87)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.86)
- PooDLe🐩: Pooled and dense self-supervised learning from naturalistic videos (ICLR 2025 · similarity 0.86)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.86)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.