Research map / Detection and segmentation / CNN features
Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a hybrid SNN (events) + ANN (frames) framework with ATW, EDS injectors and CSF fusion for energy-efficient event-based semantic segmentation, evaluated on DDD17-Seg, DSEC-Semantic and a new M3ED-Semantic.
Problem
Event-based semantic segmentation methods fail to fully exploit complementary frame and event information, and they rely on strong encoders with complex training and high computational/energy cost.
Approach
A two-branch hybrid network in which an SNN branch processes voxelized events and an ANN branch processes frames, connected by an ATW injector (temporal-weighted SNN features injected into ANN via cross attention), an EDS injector (event-guided deformable attention injecting frame features into SNN), and a channel-selection fusion module, feeding a Mask2Former decoder.
Key results
- Achieves 94.27% accuracy and 66.57% mIoU on DSEC-Semantic with 1.79M parameters and 33.45 mJ energy.
- Reaches 95.07% accuracy and 67.31% mIoU on DDD17-Seg with 9.08 mJ total energy, versus 17.89 mJ for HALSIE.
- Ablation on DSEC-Semantic: the full ATW+EDS+CSF configuration raises mIoU from 61.06% (baseline) to 66.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the hybrid frame-event method instead of OpenESS | Event-based semantic segmentation · DDD17-Seg | mIoU: 63.00 -> 67.31 | Helped |
| use the hybrid frame-event method instead of OpenESS | Event-based semantic segmentation · DSEC-Semantic | mIoU: 57.21 -> 66.57 | Helped |
| use the hybrid frame-event method instead of E2VID | Event-based semantic segmentation · M3ED-Semantic | mIoU: 39.40 -> 50.91 | Helped |
| add AWT to the frame-event baseline (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 63.59 | Helped |
| add EDS to the frame-event baseline (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 63.71 | Helped |
| add CSF to the frame-event baseline (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 62.98 | Helped |
| add AWT and EDS (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 64.37 | Helped |
| add AWT and CSF (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 64.18 | Helped |
| add EDS and CSF (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 64.36 | Helped |
| add AWT, EDS, and CSF (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 61.06 -> 66.57 | Helped |
| use 5 timesteps rather than 1 or 3 (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 64.90; 65.45; 66.57 | Helped |
| increase timesteps from 5 to 7 (ablation) | Hybrid frame-event model · DSEC-Semantic | mIoU: 66.57 -> 65.14 | Hurt |
Surprising findings
- On DDD17-Seg, EV-Segformer reports higher accuracy (94.72%) than the proposed method (95.07% is higher, but EV-Segformer has much lower mIoU of 54.41%), showing accuracy and mIoU diverge across baselines.
Limitations
- Energy is estimated from average FLOP counts rather than measured on hardware, per the text.
- Evaluated only on three driving/robotics segmentation datasets; generalization beyond them is not shown.
Experimental setup
Datasets: DDD17-Seg DSEC-Semantic M3ED-Semantic
Benchmarks: DDD17-Seg DSEC-Semantic M3ED-Semantic
Metrics: accuracy mIoU parameters GFLOPs energy (mJ)
Baselines: EV-SegNet EVDistill DTL Spiking-Deeplab E2VID EV-Transfer ViD2E ESS ESS-Sup EV-Segformer OpenESS HALSIE
Builds on: Mask2Former deformable attention Adaptor SpikingJelly
Closest papers by meaning
- ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance (AAAI 2025 · similarity 0.90)
- OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies (CVPR 2024 · similarity 0.89)
- Event-assisted Low-Light Video Object Segmentation (CVPR 2024 · similarity 0.88)
- HPL-ESS: Hybrid Pseudo-Labeling for Unsupervised Event-based Semantic Segmentation (CVPR 2024 · similarity 0.87)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.87)
- Leveraging Asynchronous Spiking Neural Networks for Ultra Efficient Event-Based Visual Processing (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.