Research map / Video / Video-language understanding
Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CERES, a plug-in causal framework that adds backdoor-adjusted language de-biasing and front-door-adjusted vision-depth mediation to pre-trained ReferFormer-style RVOS models for robust egocentric referring video object segmentation.
Problem
Ego-RVOS models learn spurious object-action correlations from skewed dataset statistics and are confounded by egocentric visual factors such as rapid motion, occlusion and perspective distortion.
Approach
Applies backdoor adjustment (NWGM approximation over object-action confounder dictionary) to the text query and front-door adjustment with a mediator built by depth-guided attention over RGB tokens (depth from Depth Anything V2) plus a memory-bank temporal context estimator.
Key results
- With ResNet101 backbone on VISOR, CERES reaches 64.0% positive mIoU (+4.1 over ActionVOS) and 72.4% gIoU.
- On VSCOS (zero-shot), CERES achieves 55.3% positive mIoU and 62.5% cIoU vs ActionVOS 52.5% and 57.7%.
- On a VISOR hard rare-concept subset, CERES improves positive mIoU from 58.4% to 62.3% over ActionVOS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LBD (backdoor language de-biasing) (ablation) | ReferFormer ResNet101 · VISOR | mIoU pos: +1.3 | Helped |
| add DAttn depth-guided mediator (ALF) for visual front-door (ablation) | ReferFormer ResNet101 · VISOR | mIoU pos: +3.4 over baseline | Helped |
| replace DAttn with nonlinear MLP depth fusion (ablation) | ReferFormer ResNet101 mediator · VISOR | mIoU neg | Hurt |
| add MAttn temporal memory attention (W=5) (ablation) | DAttn variant · VISOR | mIoU neg: lowers to 14.9% | Helped |
| add LBD to full VFE (ablation) | CERES · VISOR | mIoU neg: 15.3 vs 14.9 | Hurt |
| full CERES (LBD+VFD) vs ActionVOS | ReferFormer ResNet101 · VISOR | mIoU pos: +4.1 over ActionVOS | Helped |
| full CERES vs ActionVOS on hard rare-concept subset | ReferFormer · VISOR hard subset | mIoU pos: +3.9 | Helped |
| full CERES zero-shot vs ActionVOS | ReferFormer · VOST | mIoU pos: +1.8 | Helped |
| full CERES zero-shot vs ActionVOS | ReferFormer · VSCOS | mIoU pos: +2.8 | Helped |
| MAttn window size W=0 vs W=5 (ablation) | MAttn · VISOR | mIoU pos and Acc | Helped |
Surprising findings
- Adding an MLP-based nonlinear depth fusion gave less gain than the attention-based depth-guided mediator, and increased negative-object mIoU.
Limitations
- Confounders defined only as object-action pairs; additive score assumption is an approximation.
- Front-door validity for the chosen mediator is not formally verified; depends on pre-trained depth estimator quality.
- Sliding-window temporal attention may not capture very long-range dependencies; added computational cost.
Experimental setup
Datasets: VISOR VOST VSCOS YouTube-VOS
Benchmarks: VISOR VISOR novel subset VOST VSCOS
Metrics: mIoU cIoU gIoU Accuracy F1
Baselines: ReferFormer ReferFormer+ ActionVOS HOS EgoHOS
Builds on: ReferFormer ActionVOS Pearl front-door criterion Depth Anything V2
Closest papers by meaning
- Learning to Segment Referred Objects from Narrated Egocentric Videos (CVPR 2024 · similarity 0.89)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.88)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.87)
- Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos (ICLR 2025 · similarity 0.87)
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.