Research map / Video / Video segmentation and tracking
PooDLe🐩: Pooled and dense self-supervised learning from naturalistic videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PooDLe, a self-supervised method for naturalistic video that combines a flow-equivariant dense objective with a pooled invariance objective on flow-aligned pseudo-iconic subcrops, linked by a spatial decoder module.
Problem
Self-supervised learning on dense, cluttered, naturalistic videos (e.g., driving and first-person walking footage) suffers from semantic mismatch in pooled objectives and spatial class imbalance in dense objectives, underrepresenting small objects.
Approach
Combines a FlowE-style dense flow-equivariance loss with a pooled invariance loss over K flow-informed subcrop pairs (pseudo-iconic views), integrated via a lightweight spatial decoder module with top-down layers and UNet-style lateral connections.
Key results
- PooDLe pretrained on BDD100K reaches 39.2 mIoU on linear semantic segmentation vs. 35.7 for FlowE and 36.7 for supervised IN1K.
- With BDD initialization from IN1K weights, PooDLe reaches 44.7 mIoU on BDD linear readout, improving over its initialization by about 8 points.
- Pretrained on all Walking Tours videos, PooDLe obtains 16.5 mIoU on ADE20K linear readout vs. 13.9 for DoRA (ViT-S).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine pooled loss and the full SDM with the dense objective (ablation) | PooDLe on BDD100K · BDD100K semantic segmentation linear | mIoU: 28.8 -> 34.2 | Helped |
| add pooled loss to the dense objective without an SDM (ablation) | FlowE on BDD100K · BDD100K semantic segmentation linear | mIoU: 28.8 -> 28.9 | No clear effect |
| add a top-down decoder without lateral connections (ablation) | dense + pooled objective on BDD100K · BDD100K semantic segmentation linear | mIoU: 28.9 -> 30.3 | No clear effect |
| add lateral connections without a top-down decoder (ablation) | dense + pooled objective on BDD100K · BDD100K semantic segmentation linear | mIoU: 28.9 -> 30.3 | No clear effect |
| combine top-down decoding with lateral connections (ablation) | dense objective on BDD100K · BDD100K semantic segmentation linear | mIoU: 28.8 -> 31.8 | Helped |
| use UFlow instead of RAFT for optical flow (ablation) | PooDLe on BDD100K · BDD100K semantic segmentation linear | mIoU: 33.7 -> 34.2 | Hurt |
| initialize PooDLe from supervised ImageNet weights | PooDLe on BDD100K · BDD100K semantic segmentation linear | mIoU: 36.7 -> 44.7 | Helped |
| use PooDLe instead of FlowE | BDD100K-pretrained ResNet-50 · BDD100K semantic segmentation linear | mIoU: 35.7 -> 39.2 | Helped |
| use PooDLe instead of FlowE | BDD100K-pretrained ResNet-50 · BDD100K object detection C4 | mAP: 3.2 -> 4.9 | Helped |
| use PooDLe instead of FlowE | BDD100K-pretrained ResNet-50 · Cityscapes semantic segmentation linear | mIoU: 43.1 -> 47.2 | Helped |
| use PooDLe instead of DoRA | Walking Tours all-video pretraining · ADE20K linear | mIoU: 13.9 -> 16.5 | Helped |
| use PooDLe instead of DoRA | Walking Tours all-video pretraining · WT-Sem linear | mIoU: 7.8 -> 11.2 | Helped |
Surprising findings
- Dense SSL methods such as FlowE perform well on large, common classes but poorly on small or rare classes, a spatial imbalance effect.
- Supervised ImageNet pretraining learns small and rare classes better than BDD-pretrained dense methods, yet PooDLe trained on BDD still beats it overall on mIoU.
Limitations
- Self-supervised flow (UFlow) produces noisy results on BDD100K, though performance drops only 0.5 mIoU versus RAFT.
- Pretraining trained for fewer epochs (about 20-100) than some baselines and with a smaller ResNet-50 backbone, complicating direct comparison with DoRA.
- WT-Sem labels are generated by OpenSeeD pseudo-labels rather than human annotation.
Experimental setup
Datasets: BDD100K Walking Tours KITTI ADE20K Cityscapes ImageNet-1K
Benchmarks: BDD100K semantic segmentation BDD100K object detection Cityscapes semantic segmentation ADE20K semantic segmentation WT-Sem semantic segmentation
Metrics: mIoU pixel accuracy mAP
Baselines: DINO iBOT DoRA DenseCL PixPro FlowE MAE Supervised IN1K
Builds on: FlowE BYOL UNet FPN multi-crop
Closest papers by meaning
- Self-Supervised Learning of Video-Induced Visual Invariances (CVPR 2020 · similarity 0.86)
- Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR 2024 · similarity 0.86)
- Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · similarity 0.86)
- Self-Supervised Learning of Compressed Video Representations (ICLR 2021 · similarity 0.85)
- From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning (CVPR 2026 · similarity 0.85)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.