Research map / Video / Video segmentation and tracking
Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction
Efstathios Karypidis, Spyros Gidaris, Nikos Komodakis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hierarchical semantic forecasting stage (Re2Pix) | video diffusion baseline · Cityscapes (single-dataset training) | FID: 12.86 -> 9.90 | Improved |
| add hierarchical semantic forecasting stage (Re2Pix) | video diffusion baseline · Cityscapes (single-dataset training) | mIoU(A): 60.55 -> 63.53 | Improved |
| add REPA alignment loss | video diffusion baseline · Cityscapes (single-dataset training) | FID: 12.86 -> 12.34 | Improved |
| add VideoREPA alignment loss | video diffusion baseline · Cityscapes (single-dataset training) | FID: 12.86 -> 12.95 | No clear effect |
| increase parameters to 1.5B (Baseline-Large) | video diffusion baseline · Cityscapes (single-dataset training) | mIoU(A): 60.55 -> 61.69 | Improved |
| add nested dropout of semantic channels (ablation) | Re2Pix · Cityscapes (single-dataset training) | FID: 12.63 -> 9.90 | Improved |
| add nested dropout of semantic channels (ablation) | Re2Pix · Cityscapes (single-dataset training) | FVD: 58.91 -> 52.66 | Improved |
| add 10% predicted features to ground-truth training (mixed supervision) (ablation) | Re2Pix · Cityscapes (single-dataset training) | FVD: 80.85 -> 52.66 | Improved |
| add 10% predicted features to ground-truth training (mixed supervision) (ablation) | Re2Pix · Cityscapes (single-dataset training) | mIoU(A): 64.42 -> 63.53 | Hurt |
| add 90% ground-truth features to predicted-only training (mixed supervision) (ablation) | Re2Pix · Cityscapes (single-dataset training) | FID: 10.21 -> 9.90 | Improved |
| add 90% ground-truth features to predicted-only training (mixed supervision) (ablation) | Re2Pix · Cityscapes (single-dataset training) | mIoU(A): 62.77 -> 63.53 | Improved |
| use DINOv2 instead of SigLIP-2 as VFM (ablation) | Re2Pix · Cityscapes (single-dataset training) | mIoU(A): 63.02 -> 63.53 | Improved |
Closest papers by meaning
- Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction (ICLR 2021 · similarity 0.89)
- Learning Semantic-Aware Dynamics for Video Prediction (CVPR 2021 · similarity 0.89)
- Predicting Deeper Into the Future of Semantic Segmentation (ICCV 2017 · similarity 0.88)
- Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers (CVPR 2025 · similarity 0.88)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.88)
- Bernini: Latent Semantic Planning for Video Diffusion (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
