Research map / Video / Video segmentation and tracking
Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPSeg, a video semantic segmentation network for driving scenes that uses vanishing-point priors for explicit cross-frame motion correspondences (MotionVP), dense feature mining near the VP (DenseVP), and a context-detail fusion via contextualized motion attention.
Problem
Video semantic segmentation for driving scenes struggles with implicit cross-frame correspondences for distant small objects and fast motion, and high computational cost of processing multiple frames.
Approach
Uses vanishing point priors: static (distant objects near VP appear small, so densely partition the VP region) and dynamic (objects move radially away from the VP over time) to guide motion estimation and feature fusion within a context-detail framework with low-resolution context and high-resolution details.
Key results
- On ACDC, VPSeg (MiT-B3) reaches 77.48 mIoU, improving 2.10 points over SegFormer and 1.85 over MRCFA.
- On Cityscapes, VPSeg (MiT-B3) reaches 82.46 mIoU, improving 1.14 over SegFormer and 1.02 over CFFM.
- VPSeg adds only 2.2M parameters over SegFormer MiT-B3 (46.8M vs 44.6M) with 1124.7 GFLOPs, lower than MRCFA and CFFM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VPSeg with MiT-B3 instead of SegFormer | video semantic segmentation · ACDC val | mIoU: 75.38 -> 77.48 | Helped |
| Add VPSeg with MiT-B3 instead of MRCFA | video semantic segmentation · ACDC val | mIoU: 75.63 -> 77.48 | Helped |
| Add VPSeg with MiT-B3 instead of SegFormer | video semantic segmentation · Cityscapes val | mIoU: 81.32 -> 82.46 | Helped |
| Add VPSeg with MiT-B3 instead of CFFM | video semantic segmentation · Cityscapes val | mIoU: 81.44 -> 82.46 | Helped |
| Add DenseVP (ablation) | VPSeg · ACDC val | mIoU: 77.15 -> 77.48 | Helped |
| Add DenseVP (ablation) | VPSeg · ACDC val | mIA-IoU: 41.02 -> 41.48 | Helped |
| Use VP region a=0,b=0 rather than a=1,b=1 (ablation) | VPSeg · ACDC val | mIoU: 77.48 -> 77.28 | Hurt |
| Use VP region a=2,b=2 rather than a=1,b=1 (ablation) | VPSeg · ACDC val | mIoU: 77.48 -> 77.36 | Hurt |
| Remove the VP proximity positional embedding (ablation) | VPSeg · ACDC val | mIoU: 77.48 -> 77.02 | Hurt |
| Replace VP proximity embedding with depth map embedding (ablation) | VPSeg · ACDC val | mIA-IoU: 41.48 -> 40.78 | Hurt |
| Combine depth map and VP proximity embeddings (ablation) | VPSeg · ACDC val | mIoU: 77.48 -> 77.46 | No clear effect |
| Increase frame sampling interval from k=3 to k=5 (ablation) | VPSeg · ACDC val | mIoU: 77.48 -> 77.52 | Helped |
Surprising findings
- Replacing the VP proximity embedding with MiDaS depth map embedding gives limited mIoU gain and worse IA-IoU, with a large FPS drop.
Limitations
- VP inference adds latency, reducing FPS relative to some baselines (3.4 FPS vs 4.4 for MRCFA).
- VP detection is unstable, so the method uses a coarse VP region rather than an exact VP.
- Motion prior assumes a forward-facing camera, straight road, and linear forward motion.
Experimental setup
Datasets: ACDC Cityscapes
Benchmarks: ACDC Cityscapes
Metrics: mIoU iIoU IA-IoU FPS GFLOPs Params
Baselines: SegFormer DeepLabv3+ PSPNet OCRNet SeaFormer Video K-Net ETC TCB NetWarp CFFM MRCFA TMANet
Builds on: SegFormer DAFormer CFFM MRCFA
Closest papers by meaning
- Weakly Supervised Semantic Segmentation for Driving Scenes (AAAI 2024 · similarity 0.87)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.87)
- Scene-Centric Unsupervised Video Panoptic Segmentation (CVPR 2026 · similarity 0.86)
- Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation (CVPR 2019 · similarity 0.86)
- Cross-Dataset Collaborative Learning for Semantic Segmentation in Autonomous Driving (AAAI 2022 · similarity 0.86)
- Learning Segmentation from Point Trajectories (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.