Research map / Detection and segmentation / CNN features
Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Edge-awareness Semantic Concordance (ESC), which uses a VQ-VAE edge dictionary to re-code event and RGB features into a shared edge space with uncertainty-weighted fusion, and introduces DERS-XS, DERS-XR, and DSEC-Xtrm extreme-condition event-RGB segmentation benchmarks.
Problem
Event-RGB semantic segmentation under extreme conditions (low light, fast camera motion) suffers from feature-level mismatch between heterogeneous event and RGB modalities and from unreliable evaluation due to RGB-derived pseudo-labels.
Approach
Use a pre-trained VQ-VAE edge dictionary built from semantic boundaries as a shared discrete space; re-code image and event features into it with cross-entropy alignment, and fuse them via confidence/uncertainty-weighted attention with learnable noise embeddings.
Key results
- On DERS-XS the method reaches 67.10% mIoU, outperforming CMNeXt (64.55%) by 2.55%.
- On DERS-XR fine-tuning it outperforms EISNet by 3.41% mIoU (65.22 vs 61.81).
- Under 100x100 E-RGB masking on DERS-XS it achieves 63.87% mIoU versus 53.70% for CMNeXt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ESC instead of the strongest reported baseline, CMNeXt | Event-RGB segmentation · DERS-XS | mIoU (%): 64.55 -> 67.10; 2.55% | Helped |
| use ESC instead of the strongest reported baseline, EISNet | Event-RGB segmentation · DERS-XR | mIoU (%): 61.81 -> 65.22; 3.41% | Helped |
| use ESC instead of the strongest reported baseline, CMNeXt | Event-RGB segmentation · DSEC-Semantic | mIoU (%): 69.03 -> 71.04; 2.01% | Helped |
| use ESC instead of the strongest reported baseline, EISNet | Event-RGB segmentation · DSEC-Xtrm | mIoU (%): 48.76 -> 50.87; 2.11% | Helped |
| remove upsampling (ablation) | ESC · DERS-XS, no mask | mIoU (%): 67.10 -> 64.59 | Hurt |
| remove RC (ablation) | ESC · DERS-XS | mIoU (%): 67.10 -> 64.29 (no mask); 63.87 -> 59.53 (E-RGB masking) | Hurt |
| remove UO (ablation) | ESC · DERS-XS | mIoU (%): 67.10 -> 62.53 (no mask); 63.87 -> 58.43 (E-RGB masking) | Hurt |
| remove ELR and L_edge (ablation) | ESC · DERS-XS | mIoU (%): 67.10 -> 61.35 (no mask); 63.87 -> 56.34 (E-RGB masking) | Hurt |
| vary edge dictionary size from K=128 to K=16, 32, 64, 256, or 512 (ablation) | ESC · DERS-XS | mIoU (%): 67.10 -> 65.87 (K=16), 66.84 (K=32), 66.64 (K=64), 66.54 (K=256), 66.11 (K=512) | Hurt |
| remove noise embeddings N_K and N_V (ablation) | ESC · DERS-XS | mIoU (%): 67.10 -> 66.05; 1.05% | Hurt |
| remove noise embeddings N_K and N_V (ablation) | ESC · DSEC-Semantic | mIoU (%): 71.04 -> 70.86; 0.18% | No clear effect |
| remove noise embeddings N_K and N_V (ablation) | ESC · DSEC-Xtrm | mIoU (%): 50.87 -> 50.05; 0.81% | Hurt |
Surprising findings
- Removing noise embeddings yields only a 0.18% mIoU drop on DSEC-Semantic, attributed to RGB-based pseudo-label bias favoring RGB.
- Reduced ESC variant with 2x MiT-B0 backbones beats CMNeXt (49.06 vs 45.16 mIoU on DSEC-Xtrm) with fewer parameters.
Limitations
- Only event-RGB fusion is considered; other visual modalities are not explored.
- FLOPs are relatively high due to multiple MLP heads for re-coding and fusion.
- DERS-XR has only 120 annotated test frames and coarse labels.
Experimental setup
Datasets: DERS-XS DERS-XR DSEC-Semantic DSEC-Xtrm CARLA DAVIS346
Benchmarks: DERS-XS DERS-XR DSEC-Semantic DSEC-Xtrm
Metrics: mIoU mACC gACC
Baselines: SegFormer EvSegFormer TokenFusion CMX CMNeXt EISNet
Builds on: VQ-VAE SegFormer CMX CMNeXt ESS
Closest papers by meaning
- ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance (AAAI 2025 · similarity 0.89)
- Keep the Balance: A Parameter-Efficient Symmetrical Framework for RGB+X Semantic Segmentation (CVPR 2025 · similarity 0.86)
- RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.85)
- Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach (AAAI 2025 · similarity 0.85)
- The Shape of Events: Edge-Based Inductive Biases via Cross-Domain Distillation (NeurIPS 2026 · similarity 0.85)
- DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.