Research map / Detection and segmentation / Open-set and few-shot detection
EZSR: Event-based Zero-Shot Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an event encoder (initialized from EVA-02-CLIP ViT-B/16) to align event embeddings with frozen CLIP image embeddings using a contrastive loss plus a scalar-wise modulation loss, trained on ~180M synthetic event frames generated from static RGB images via affine motion and V2E.
Problem
Zero-shot object recognition on event camera data is hampered by spatially sparse event embeddings that cause misalignment between event and CLIP text embedding spaces, and by scarce paired event-RGB training data.
Approach
Adds a scalar-wise modulation loss that penalizes per-dimension deviations between event and paired RGB embeddings, weighted by how similar each event embedding is to other events, plus a synthetic event data pipeline that warps static RGB images with random translation, scaling, and rotation and converts them to events with V2E.
Key results
- Achieves 47.84% zero-shot accuracy on N-ImageNet with ViT-B/16, up from 9.57% for the contrastive baseline (48.86% with both components).
- Reaches 90.44% on N-Caltech101 and 64.24% on HARDVS zero-shot, outperforming other zero-shot frameworks such as EventBind and EventCLIP.
- Scaling the CLIP image encoder to ViT-bigE/14+ (EVA) raises N-ImageNet zero-shot accuracy to 62.28%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scalar-wise modulation to the baseline objective (ablation) | EZSR event encoder · N-ImageNet | accuracy: 9.57 -> 47.84 | Helped |
| Add reference-dataset embedding translation (Remark 1) to the baseline (ablation) | EZSR event encoder · N-ImageNet | accuracy: 9.57 -> 43.48 | Helped |
| Add Remark 1 embedding translation to scalar-wise modulation (ablation) | EZSR event encoder · N-ImageNet | accuracy: 47.84 -> 48.63 | Helped |
| Remove the baseline contrastive loss while retaining scalar-wise modulation (ablation) | EZSR event encoder · N-ImageNet | accuracy: 48.86 -> 47.80 | No clear effect |
| Train on the in-domain N-ImageNet training set instead of the synthetic dataset (ablation) | EZSR event encoder · N-ImageNet | accuracy: 47.84% vs. 54.36% | Helped |
| Fine-tune using synthetic event data and CC3M text data (ablation) | EZSR event encoder · N-ImageNet | accuracy: 47.84% -> 14.84% | Hurt |
| Increase the number of synthetic training examples (ablation) | EZSR event encoder · N-ImageNet | accuracy | Helped |
| Train for more epochs, up to the reported saturation point (ablation) | EZSR event encoder · N-ImageNet | accuracy | Helped |
| Scale the CLIP image-encoder backbone and inference computation (ablation) | EZSR event encoder · N-ImageNet | accuracy | Helped |
Surprising findings
- Training on the out-of-domain synthetic data (47.84%) comes close to training on the N-ImageNet training set itself (54.36%).
- Fine-tuning the synthetic-data-trained network on CC3M text data decreased zero-shot accuracy to 14.84%.
Limitations
- Gap remains toward a generalist event foundation model.
- Lower performance on low-resolution datasets such as CIFAR-10-DVS and N-MINIST, falling behind dataset-specific methods.
- Zero-shot action recognition requires an extra trained two-layer ViT for temporal aggregation.
Experimental setup
Datasets: synthetic event data from RGB images (around 180M RGB images from multiple sources) N-ImageNet CC3M
Benchmarks: N-ImageNet N-Caltech101 CIFAR-10-DVS N-MINIST HARDVS DailyAction PAF Bullying10K UCF101-DVS
Metrics: accuracy
Baselines: EventBind ECLIP EventCLIP ECDP MEM EARSNN PSN
Builds on: CLIP InfoNCE EventBind EventCLIP V2E
Closest papers by meaning
- PASS: Path-selective State Space Model for Event-based Recognition (NeurIPS 2025 · similarity 0.84)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.84)
- Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition (NeurIPS 2025 · similarity 0.84)
- FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies (NeurIPS 2025 · similarity 0.83)
- Transductive Zero-Shot and Few-Shot CLIP (CVPR 2024 · similarity 0.83)
- EGSST: Event-based Graph Spatiotemporal Sensitive Transformer for Object Detection (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.