Research map / Label-efficient and robust learning / Self-supervised and contrastive
DINO-Foresight: Looking into the Future with DINO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DINO-Foresight, which forecasts future frames' DINOv2 features with a masked feature transformer trained self-supervised, then applies off-the-shelf task heads (segmentation, instance segmentation, depth, normals) to predicted features.
Problem
Future prediction for autonomous driving and robotics is usually done by generating pixel-level RGB frames or VAE latents, which is expensive and spends capacity on low-level details rather than semantics.
Approach
Forecast the temporal evolution of multi-layer, PCA-reduced frozen VFM (DINOv2) features with a masked transformer, so the predicted feature space serves as a latent on which modular task heads attach without retraining the predictor.
Key results
- On Cityscapes short-term semantic segmentation, DINO-Foresight reaches 71.8 ALL mIoU vs 65.9-73.9 for prior methods and 64.9 for fine-tuned VISTA, with 88.6 delta1 depth and 94.4 surface normal accuracy at 11.25°.
- Forecasting VFM features is far cheaper than VISTA: mid-term forecasting on 500 Cityscapes val scenes takes about 5 minutes vs 8.3 hours on one A100.
- Zero-shot transfer from Cityscapes to nuScenes is only slightly worse than in-domain training and still beats VISTA fine-tuned on nuScenes (depth delta1 78.4 vs 74.6 at mid-term).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Forecast DINOv2 features instead of generating future RGB frames | DINO-Foresight vs VISTA_ft · Cityscapes short semantic segmentation (ALL) | mIoU: 64.9 -> 71.8 | Helped |
| Forecast DINOv2 features instead of generating future RGB frames | DINO-Foresight vs VISTA_ft · Cityscapes short instance segmentation | AP50: 33.1 -> 50.5 | Helped |
| Train at 224 then fine-tune at 448 (ablation) | DINO-Foresight high-resolution forecasting · Cityscapes short semantic segmentation (ALL) | mIoU: 64.34 -> 71.81 | Helped |
| Use sliding-window crops for high-resolution forecasting (ablation) | DINO-Foresight high-resolution forecasting · Cityscapes short semantic segmentation (ALL) | mIoU: 64.34 -> 71.26 | Helped |
| Apply PCA dimensionality reduction (ablation) | DINO-Foresight · Cityscapes short depth | δ1: 87.9 -> 88.6 | Helped |
| Use multi-layer DINO features from layers 3, 6, 9, and 12 (ablation) | DINO+DPT · Cityscapes semantic segmentation (ALL) | mIoU: 72.1 -> 77.0 | Helped |
| Mask all future features rather than use random masking (ablation) | DINO-Foresight · Cityscapes short semantic segmentation (ALL) | mIoU: 70.5 -> 71.8 | Helped |
| Replace SmoothL1 with L1, MSE, or SmoothL1+Cos (ablation) | DINO-Foresight · Cityscapes semantic segmentation and dense prediction tasks | mIoU: 71.7; 71.7; 71.7; 71.8 | No clear effect |
| Use continuous DINOv2 features rather than discrete tokenizer codes (ablation) | DINO-Foresight · Cityscapes short semantic segmentation (ALL) | mIoU: 61.7 -> 68.9 | Helped |
| Increase model size from Small (115M) to Large (460M) (ablation) | DINO-Foresight · Cityscapes mid semantic segmentation (ALL) | mIoU: 59.3 -> 60.2 | Helped |
| Add nuScenes to Cityscapes training data (ablation) | DINO-Foresight · Cityscapes mid semantic segmentation (ALL) | mIoU: 59.8 -> 61.0 | Helped |
| Evaluate Cityscapes-trained DINO-Foresight on nuScenes without fine-tuning | DINO-Foresight · nuScenes mid depth | δ1: 80.7 -> 78.4 | Hurt |
Surprising findings
- Continuous VFM feature forecasting outperforms a discrete 4M DINOv2 tokenizer variant despite comparable oracle performance.
- Previously predicted-feature heads sometimes outperform the Oracle on some metrics? No; instead, intermediate transformer layers of the forecaster slightly improve downstream DPT segmentation and depth.
Limitations
- Current implementation is deterministic, so it does not model future uncertainty.
- Inference-time compute remains high at high feature resolution.
- Evaluated only on driving datasets (Cityscapes, nuScenes) with relatively short horizons.
Experimental setup
Datasets: Cityscapes nuScenes
Benchmarks: Cityscapes nuScenes
Metrics: mIoU (ALL) MO-mIoU AP50 AP depth delta1 AbsRel surface normal mean angular error 11.25 degree accuracy
Baselines: Oracle Copy-Last VISTA Futurist PFA F2MF APANet DeformF2F
Builds on: DINOv2 DPT Mask2Former VISTA
Closest papers by meaning
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.88)
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (NeurIPS 2025 · similarity 0.87)
- Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers (CVPR 2025 · similarity 0.87)
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (ICML 2025 · similarity 0.87)
- Visual Point Cloud Forecasting enables Scalable Autonomous Driving (CVPR 2024 · similarity 0.85)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.