Research map / 3D vision / NeRF and radiance fields
Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fuses 2D motion segmentation masks (Motion Grouping) into layered NeRF (NeuralDiff) via positive/negative motion fusion losses plus test-time refinement on selected frames, improving unsupervised dynamic object segmentation on EPIC Fields.
Problem
Unsupervised segmentation of dynamic and semi-static objects in long egocentric videos, where 3D neural radiance field methods lag behind 2D motion segmentation baselines.
Approach
Lift sparse, precise 2D motion-segmentation masks into a layered radiance field by pulling the dynamic layer toward the 2D masks (positive fusion) and pushing the semi-static layer away from them (negative fusion), then refine the dynamic and semi-static layers at test time on selected frames to reduce scene complexity.
Key results
- Motion fusion with test-time refinement on NeuralDiff raises dynamic-object mAP from 55.58 to 72.51 (+30.4%) on EPIC Fields UDOS, outperforming the 2D Motion Grouping baseline (64.27).
- Semi-static mAP improves from 25.55 to 27.70 (+8.4%) and dynamic+semi-static mAP from 69.74 to 74.21 (+6.4%).
- Applying the fusion to EgoHOS raises hand-object mAP from 71.20 to 77.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add test-time refinement and layered motion fusion to NeuralDiff | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 72.51 | Helped |
| Add test-time refinement and layered motion fusion to NeuralDiff | NeuralDiff · EPIC Fields UDOS | mAP SS: 25.55 -> 27.70 | Helped |
| Add test-time refinement and layered motion fusion to NeuralDiff | NeuralDiff · EPIC Fields UDOS | mAP Dyn+SS: 69.74 -> 74.21 | Helped |
| Add test-time refinement and positive motion fusion | NeRF-W · EPIC Fields UDOS | mAP Dyn: 28.52 -> 34.20 (19.9%) | Helped |
| Add test-time refinement and positive motion fusion | NeRF-T · EPIC Fields UDOS | mAP Dyn: 44.27 -> 51.11 (15.4%) | Helped |
| Add test-time refinement and positive motion fusion | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 67.23 (20.9%) | Helped |
| Apply the proposed method to MG | MG · Dynamic hand-object segmentation | mAP: 64.27 -> 72.51 | Helped |
| Apply the proposed method to EgoHOS | EgoHOS · Dynamic hand-object segmentation | mAP: 71.20 -> 77.31 | Helped |
| Add negative motion fusion (ablation) | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 63.73 | Helped |
| Add positive motion fusion (ablation) | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 63.38 | Helped |
| Add both positive and negative motion fusion (ablation) | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 66.14 | Helped |
| Add test-time refinement (ablation) | NeuralDiff · EPIC Fields UDOS | mAP Dyn: 55.58 -> 62.29 | Helped |
Surprising findings
- Adding temporal neighbouring frames during test-time refinement gave no benefit over using only the selected reference frames.
Limitations
- Gains depend on the quality of the 2D motion segmentation teacher; partial masks are fused.
- Test-time refinement adds per-scene optimization cost (about 13 s per frame).
Experimental setup
Datasets: EPIC-KITCHENS EPIC Fields
Benchmarks: EPIC Fields UDOS
Metrics: mean average precision (mAP)
Baselines: Motion Grouping (MG) NeRF-W NeRF-T NeuralDiff EgoHOS
Builds on: NeuralDiff Motion Grouping EPIC Fields Semantic NeRF SfM-TTR
Closest papers by meaning
- DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videos (CVPR 2025 · similarity 0.88)
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025 · similarity 0.88)
- Scalable Benchmarking and Robust Learning for Noise-Free Ego-Motion and 3D Reconstruction from Noisy Video (ICLR 2025 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.86)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.