Research map / Video / Video segmentation and tracking
Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semantic Flow, a neural semantic field for dynamic monocular video scenes that learns semantics from 3D flow features (rather than per-point features) with volume densities as opacity priors, enabling editing, completion, tracking, and adaptation.
Problem
Learning semantic fields of dynamic scenes from monocular videos suffers from 2D-to-3D ambiguity in the viewing direction; point-based dynamic NeRFs with semantic heads overfit to training views and generalize poorly with few labels or on novel scenes.
Approach
Learn semantic logits from continuous flow features (sampled from video frames along predicted point trajectories, refined by flow attention) and render them along rays with NeRF volume densities as opacity priors, so 2D frame labels supervise the 3D flow semantics.
Key results
- On multi-scene semantic learning, Semantic Flow reaches 0.844 mIoU on Balloon1 versus 0.760 for MonoNeRF with semantic head and 0.515 for DynNeRF with semantic head.
- With 50% labels for semantic completion, Semantic Flow achieves 0.961 total accuracy and 0.818 mIoU, above MonoNeRF (0.956 / 0.786) and DynNeRF (0.934 / 0.738).
- In dynamic scene tracking with 25% labels, Semantic Flow reaches 0.942 total accuracy and 0.767 mIoU, versus 0.935 / 0.716 for MonoNeRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn semantics from flow features rather than point features | Semantic Flow versus MonoNeRF + semantic head · Balloon1, Balloon2, Jumping, and Skating (multiple-scene learning) | mIoU: 0.760 -> 0.844; 0.616 -> 0.839; 0.576 -> 0.703; 0.590 -> 0.608 | Helped |
| Learn semantics from flow features rather than point features | Semantic Flow versus strongest baseline · Umbrella, Playground, and Truck (novel-scene adaptation, 500 steps) | mIoU: 0.795 -> 0.884; 0.387 -> 0.742; 0.662 -> 0.765 | Helped |
| Use Semantic Flow instead of MonoNeRF + semantic head | Semantic Flow · Semantic completion (50% labels, average) | mIoU: 0.786 -> 0.818 | Helped |
| Use Semantic Flow instead of MonoNeRF + semantic head | Semantic Flow · Dynamic scene tracking (25% labels, average) | mIoU: 0.716 -> 0.767 | Helped |
| Add the σ_dy opacity prior (ablation) | Semantic Flow · Multiple-scene learning | mIoU: 0.838 -> 0.859 | Helped |
| Add the flow attention module (ablation) | Semantic Flow · Multiple-scene learning | mIoU: 0.643 -> 0.859 | Helped |
| Add the static-scene semantic network Ψ_st^sem (ablation) | Semantic Flow · Multiple-scene learning | mIoU: 0.734 -> 0.859 | Helped |
| Add the semantic consistency loss L_consist (ablation) | Semantic Flow · Multiple-scene learning | mIoU: 0.741 -> 0.859 | Helped |
| Add relative flow displacement ΔΓ (ablation) | Semantic Flow · Multiple-scene learning | mIoU: 0.770 -> 0.859 | Helped |
Surprising findings
- Using FlowNet instead of RAFT for supervision drops mIoU by about 10% on the Jumping and Skating scenes, but Balloon1 is slightly improved, suggesting small flow noise can help.
Limitations
- Predictions degrade when the pretrained optical flow estimates are imperfect, as in the tracking failure cases.
- Training semantic fields with very few labels remains challenging; the method fails on some occluded regions where flow is wrong.
- Evaluated on only seven annotated forward-facing scenes from a single dataset.
Experimental setup
Datasets: Dynamic Scene dataset (Yoon et al.) Kinetics-400 ImageNet
Benchmarks: Semantic Dynamic Scene dataset
Metrics: Total Acc mIoU Avg Acc PSNR SSIM LPIPS
Baselines: DynNeRF with semantic head MonoNeRF with semantic head DeAOT NeRF+time NSFF RoDyNeRF D-NeRF Masked R-CNN
Builds on: DynNeRF MonoNeRF Semantic NeRF SlowOnly RAFT
Closest papers by meaning
- Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance (NeurIPS 2025 · similarity 0.88)
- DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method (NeurIPS 2025 · similarity 0.88)
- FlowFeat: Pixel-Dense Embedding of Motion Profiles (NeurIPS 2025 · similarity 0.87)
- Flow-NeRF: Joint Learning of Geometry, Poses, and Dense Flow within Unified Neural Representations (CVPR 2025 · similarity 0.86)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
- H-Flow: Self-supervised Human Scene Flow via Physics-inspired Joint Multi-modal Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.