Research map / Vision-language and multimodal / Human and embodied modeling
Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Talk2Event, a large-scale event-camera referring-expression grounding benchmark built on DSEC driving data, and EventRefer, an attribute-aware grounding framework with a mixture of event-attribute experts.
Problem
Language-driven object grounding has been studied for RGB and 3D scenes but not for event camera streams, which are asynchronous and motion-sensitive; no benchmark links event data to free-form referring expressions with bounding boxes.
Approach
Annotate referring expressions with four grounding attributes (appearance, status, relation-to-viewer, relation-to-others) generated from temporal context frames, and fuse attribute-specific event-text features via a mixture of event-attribute experts with multi-attribute pseudo-target matching and late score fusion at inference.
Key results
- EventRefer reaches 31.96% mAcc in event-only setting vs. 29.34% for the best event baseline (EvRT-DETR) on Talk2Event val.
- EventRefer reaches 61.82% mAcc in event-frame fusion vs. 59.40% for FlexEvent.
- In frame-only setting EventRefer reaches 55.47% mAcc vs. 48.91% for BUTD-DETR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add positive word matching (PWM) (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 22.07% -> 26.38% | Helped |
| Add multi-attribute fusion (MAF) (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 22.07% -> 27.01% | Helped |
| Combine PWM and MAF (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 22.07% -> 29.66% | Helped |
| Add MoEE to PWM and MAF (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 29.66% -> 31.96% | Helped |
| Use status rather than appearance alone (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 27.98% -> 28.90% | Helped |
| Use relation-to-viewer alone rather than appearance alone (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 27.98% -> 27.03% | Hurt |
| Use relation-to-others alone rather than appearance alone (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 27.98% -> 26.97% | Hurt |
| Use all four attributes jointly rather than appearance alone (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 27.98% -> 31.96% | Helped |
| Replace no fusion with additive fusion (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 26.38% -> 28.39% | Helped |
| Replace no fusion with concatenation (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 26.38% -> 27.50% | Helped |
| Replace no fusion with attention fusion (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 26.38% -> 29.66% | Helped |
| Replace attention fusion with MoEE (ablation) | EventRefer · Talk2Event val, event-only | mAcc: 29.66% -> 31.96% | Helped |
Surprising findings
- Viewer-relation and other-object-relation attributes alone slightly lower mAcc than the appearance-only baseline, though combining all four gives the best result.
Limitations
- Event-only performance remains well below frame-based and fusion settings.
- Annotations are generated by a VLM from driving scenes in one dataset (DSEC, Switzerland), limiting diversity.
Experimental setup
Datasets: DSEC Talk2Event
Benchmarks: Talk2Event
Metrics: mAcc (IoU@0.95 top-1 accuracy) mIoU
Baselines: MDETR BUTD-DETR OWL-ViT OWL-v2 YOLO-World GroundingDINO RVT LEOD SAST SSMS EvRT-DETR RENet
Builds on: RVT DETR BUTD-DETR FlexEvent RoBERTa
Closest papers by meaning
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.89)
- Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- EventPrune: Cascaded Event-Assisted Token Pruning for Efficient First-Person Dynamic Spatial Reasoning (NeurIPS 2026 · similarity 0.87)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.87)
- OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.