Research map / Video / Video segmentation and tracking
Know Where You Are From: Event-Based Segmentation via Spatio-Temporal Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KWYAF, an event-based semantic segmentation framework that injects historical spatio-temporal motion cues via Sequential Motion Encoding modules and an Event-based Reliable Region Selection Mechanism, reaching SOTA on DDD17 and DSEC.
Problem
Event-based semantic segmentation struggles to distinguish objects with similar contours and to segment spatially continuous objects, because event data is sparse, lacks color, and triggers only at motion edges.
Approach
Embed historical motion states of objects into the segmented scene by correlating features across a sequence of event voxel grids (SME modules), and filter the correlation volume with local window masking and event-density-based reliable region selection (ER2SM), before decoding with a CFM head.
Key results
- On DDD17, KWYAF reaches 57.69% mIoU with MiT-B0 (90.04% Acc), 1.02 points above ESS-Sup and 0.66 above CFFM++.
- On DSEC-Semantic, KWYAF reaches 57.75% mIoU and 90.87% Acc, outperforming CFFM++ (56.75%) and HMNet (55.00%).
- Ablation on DSEC: SME alone raises mIoU from 54.19 to 56.93; adding ER2SM gives a further 0.82 points to 57.75.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KWYAF spatio-temporal propagation to event-based segmentation | DDD17 · DDD17 | mIoU: 56.67 -> 57.69 | Helped |
| Add KWYAF spatio-temporal propagation to event-based segmentation | DSEC · DSEC | mIoU: 56.75 -> 57.75 | Helped |
| Add ER2SM to SME (ablation) | KWYAF · DSEC | mIoU: 56.93 -> 57.75 | Helped |
| Add SME and ER2SM (ablation) | DSEC baseline · DSEC | mIoU: 54.19 -> 57.75 | Helped |
| Replace the proposed motion embedding with design A (ablation) | KWYAF · DSEC | mIoU: 56.38 -> 57.75 | Hurt |
| Replace the proposed motion embedding with design B (ablation) | KWYAF · DSEC | mIoU: 57.46 -> 57.75 | Hurt |
| Add LWM with size 3 (ablation) | KWYAF ER2SM · DSEC | mIoU: 56.93 -> 57.33 | Helped |
| Add RRS to LWM size 3 (ablation) | KWYAF ER2SM · DSEC | mIoU: 57.33 -> 57.16 | Hurt |
| Use LWM size 7 without RRS (ablation) | KWYAF ER2SM · DSEC | mIoU: 57.40 -> 57.75 | Hurt |
| Add RRS to LWM size 7 (ablation) | KWYAF ER2SM · DSEC | mIoU: 57.40 -> 57.75 | Helped |
| Use LWM size 11 without RRS (ablation) | KWYAF ER2SM · DSEC | mIoU: 57.29 -> 57.75 | Hurt |
| Add RRS to LWM size 11 (ablation) | KWYAF ER2SM · DSEC | mIoU: 57.29 -> 57.39 | Helped |
Surprising findings
- EvDistill, which uses RGB knowledge transfer with a much larger model, slightly outperforms KWYAF on DDD17 (58.02 vs 57.69 mIoU).
Limitations
- Small local window size reduces the benefit of reliable region selection, so performance depends on window size tuning (m=7 chosen empirically).
- Gains over the best VSS and RGB-distillation baselines are modest (under 1 mIoU point on DDD17).
Experimental setup
Datasets: DDD17 DSEC-Semantic
Benchmarks: DDD17 DSEC-Semantic
Metrics: mIoU Acc
Baselines: Ev-SegNet E2VID Vid2E EvDistill ESS ESS-Sup EvSegFormer HMNet SpikingEDN SegFormer CFFM MRCFA
Builds on: SegFormer CFFM RAFT correlation volume ESS-Sup EvDistill
Closest papers by meaning
- Segment Any Motion in Videos (CVPR 2025 · similarity 0.88)
- Learning Segmentation from Point Trajectories (NeurIPS 2024 · similarity 0.86)
- ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance (AAAI 2025 · similarity 0.86)
- Event-assisted Low-Light Video Object Segmentation (CVPR 2024 · similarity 0.86)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.86)
- GMOS: Grounding Moving Object Segmentation in 3D Space and Time (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.