Research map / Video / Video-language understanding
DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIV-FF, a three-stream dynamic neural feature field that distills both CLIP image-language and EgoVideo video-language features to decompose egocentric scenes into persistent, dynamic, and actor components.
Problem
Environment understanding from egocentric videos, where the camera wearer's continuous motion and interactions with dynamic objects break the rigid-scene assumption of existing language feature fields, and where single-image semantics miss action/affordance information.
Approach
Extends a three-stream dynamic radiance field (NeuralDiff) with image-language (CLIP, SAM-mask-aligned) and video-language (EgoVideo, patch and hotspot-masked global) feature fields, so persistent, dynamic and actor components each carry semantics, with frame-specific codes for temporal consistency.
Key results
- Full DIV-FF with image inference reaches 30.5 mIoU on dynamic object segmentation vs 21.7 for NeuralDiff+OWL-ViT+SAM (+40.5%) and 12.8 for LERF.
- Video-language field reaches 20.7 mIoU on affordance segmentation vs 12.2 for LERF (+69.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIV-FF with image-language features and the full model | DIV-FF · EPIC-Diff, dynamic object segmentation | mIoU: 21.7 -> 30.5 | Helped |
| Use dynamic geometry and CLIP patch features (ablation) | LERF · EPIC-Diff, dynamic object segmentation | mIoU: 12.8 -> 19.8 | Helped |
| Replace CLIP patch features with SAM-mask features (ablation) | DIV-FF · EPIC-Diff, dynamic object segmentation | mIoU: 19.8 -> 26.2 | Helped |
| Add bounding-box features to the SAM-mask CLIP descriptor (ablation) | DIV-FF · EPIC-Diff, dynamic object segmentation | mIoU: 26.2 -> 30.5 | Helped |
| Render from the video-language feature field instead of the image-language field (ablation) | DIV-FF · EPIC-Diff, dynamic object segmentation | mIoU: 30.5 -> 16.6 | Hurt |
| Use the video-language feature field for affordance segmentation | DIV-FF · EPIC-Diff, affordance segmentation | mIoU: 12.2 -> 20.7 | Helped |
| Use CLIP patch features instead of the video-language feature field (ablation) | DIV-FF · EPIC-Diff, affordance segmentation | mIoU: 20.7 -> 13.3 | Hurt |
| Render from the image-language field instead of the video-language field (ablation) | DIV-FF · EPIC-Diff, affordance segmentation | mIoU: 20.7 -> 11.8 | Hurt |
| Add global video-feature supervision in the interaction hotspot mask to local patch-level supervision (ablation) | DIV-FF video-language feature field · EPIC-Diff, affordance relevancy maps | relevancy map quality | Helped |
Surprising findings
- For single-image models, patch-based CLIP features (LERF, CLIP in patches) outperform the full model that uses object masks on affordance segmentation.
Limitations
- Evaluation limited to 10 sequences of one dataset subset (EPIC-Diff).
- Full model underperforms on affordance segmentation with image-language features; video-language field needed.
- Relies on pre-extracted hand-object detector masks for the video-language hotspot supervision.
Experimental setup
Datasets: EPIC-Diff (EPIC-Kitchens subset) EPIC-Kitchens
Benchmarks: EPIC-Diff dynamic object segmentation annotations (from N3F) EPIC-Diff affordance segmentation (manually annotated ~700 masks)
Metrics: mIoU
Baselines: LERF NeuralDiff + OWL-ViT NeuralDiff + OWL-ViT + SAM OWL-ViT OWL-ViT + SAM
Builds on: NeuralDiff LERF EgoVideo CLIP Segment Anything Model
Closest papers by meaning
- Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videos (CVPR 2025 · similarity 0.88)
- SAW-Bench: Learning Situated Awareness in the Real World (ICML 2026 · similarity 0.86)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.86)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.86)
- EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretraining (NeurIPS 2025 · similarity 0.85)
- Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.