Research map / Vision-language and multimodal / CLIP and zero-shot
OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills CLIP knowledge from image-text pairs into event-camera encoders via frame-to-event contrastive distillation and text-to-event consistency regularization, enabling open-vocabulary, annotation-free event semantic segmentation.
Problem
Event-based semantic segmentation is limited by costly dense event annotations and closed label sets; existing image-to-event domain adaptation has representational gaps.
Approach
Transfer frozen CLIP image/text knowledge to event streams using superpixel-driven frame-to-event contrastive distillation (superpixels from SLIC or SAM) plus a text-guided event-text consistency loss that mitigates intra-class conflicts in the contrastive objective.
Key results
- Annotation-free OpenESS reaches 53.93% mIoU on DDD17-Seg and 43.31% mIoU on DSEC-Semantic, well above MaskCLIP and FC-CLIP.
- Open-vocabulary OpenESS achieves 63.00% mIoU on DDD17-Seg and 57.21% on DSEC-Semantic, exceeding prior fully-supervised methods such as HALSIE (60.66 / 52.43).
- In few-shot 1% fine-tuning on DSEC-Semantic with voxel grids, OpenESS reaches 41.41 mIoU versus 26.62 for random initialization, with >30% relative gains under linear probing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer CLIP knowledge to events using F2E and T2E | OpenESS · DDD17-Seg test, annotation-free | mIoU: 51.12 -> 53.93 | Helped |
| transfer CLIP knowledge to events using F2E and T2E | OpenESS · DSEC-Semantic test, annotation-free | mIoU: 39.42 -> 43.31 | Helped |
| use OpenESS for open-vocabulary segmentation | DDD17-Seg · DDD17-Seg test, open-vocabulary ESS | mIoU: 62.01 -> 63.00 | Helped |
| use OpenESS for open-vocabulary segmentation | DSEC-Semantic · DSEC-Semantic test, open-vocabulary ESS | mIoU: 55.67 -> 57.21 | Helped |
| add F2E contrastive distillation (ablation) | OpenESS frame2voxel without T2E · DDD17-Seg test, voxel grid, linear probing | mIoU: 12.30 -> 52.60 | Helped |
| add T2E semantic regularization (ablation) | OpenESS frame2voxel without F2E · DDD17-Seg test, voxel grid, linear probing | mIoU: 12.30 -> 54.11 | Helped |
| add F2E to a model with T2E (ablation) | OpenESS frame2voxel · DDD17-Seg test, voxel grid, linear probing | mIoU: 54.11 -> 55.61 | Helped |
| add T2E to a model with F2E (ablation) | OpenESS frame2voxel · DDD17-Seg test, voxel grid, linear probing | mIoU: 52.60 -> 55.61 | Helped |
| add F2E to a model with T2E (ablation) | OpenESS frame2recon · DDD17-Seg test, reconstruction, linear probing | mIoU: 52.62 -> 53.61 | Helped |
| add T2E to a model with F2E (ablation) | OpenESS frame2recon · DDD17-Seg test, reconstruction, linear probing | mIoU: 50.21 -> 53.61 | Helped |
| add F2E to a model with T2E (ablation) | OpenESS frame2spike · DDD17-Seg test, spike, linear probing | mIoU: 16.11 -> 16.27 | Helped |
| add T2E to a model with F2E (ablation) | OpenESS frame2spike · DDD17-Seg test, spike, linear probing | mIoU: 15.07 -> 16.27 | Helped |
Surprising findings
- Pre-training on out-of-distribution event data (cross-dataset) still gives clear gains over random initialization.
- Event-only pre-training using E2VID reconstructions as the teacher input is clearly weaker than frame-based distillation.
Limitations
- Heuristic superpixels (SLIC) give only coarse grouping, causing intra-class optimization conflicts in the contrastive loss.
- Event-only variant using reconstructions performs worse than frame-based distillation.
- Superpixel count hyperparameter must be tuned per dataset.
Experimental setup
Datasets: DDD17-Seg DSEC-Semantic
Benchmarks: DDD17-Seg DSEC-Semantic
Metrics: mIoU Acc
Baselines: MaskCLIP FC-CLIP Ev-SegNet E2VID Vid2E EVDistill DTL ESS HALSIE HMNet
Builds on: CLIP SAM SLIC E2VID MaskCLIP
Closest papers by meaning
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.89)
- Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approach (AAAI 2025 · similarity 0.89)
- ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance (AAAI 2025 · similarity 0.88)
- Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.87)
- LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction (NeurIPS 2024 · similarity 0.86)
- RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.