Research map / Vision-language and multimodal / Multimodal LLMs

EventGPT: Event Stream Understanding with Multimodal Large Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EventGPT, the first MLLM for event stream understanding, built from an event encoder, spatio-temporal aggregator, projector, event-language adapter and Vicuna-based LLM, trained in three stages on image-text, synthetic event-text, and real event instruction data.

Problem

Existing MLLMs are built for natural RGB images and degrade in scenarios (low light, high dynamic range, high-speed motion) where event cameras are better suited; there is no LLM-based model for event stream understanding and the domain gap between events and language is large.

Approach

A three-stage training paradigm that first aligns a linear projector using RGB image-text data (LLaVA-Pretrain), then trains a spatio-temporal aggregator and event-language adapter on a large synthetic event-text set (N-ImageNet-Chat), and finally fully fine-tunes on real event instruction data (Event-Chat), with the LLM serving as an anchor.

Key results

• EventGPT-7B scores 3.52 DC, 4.09 CR, 4.29 VQA on Event-Chat, versus 2.20-2.41 DC for the 7B baselines (LLaVA, Qwen2-VL, InternVL2, DeepSeek-VL).
• EventGPT-13B reaches 2.41 DC, 2.81 CR, 2.40 VQA on N-ImageNet-Chat, the best among compared models there.
• Ablation: adding the event-language adapter improves Event-Chat DC from 3.40 to 3.48 and the spatio-temporal aggregator to 3.51; full model reaches 3.52.

Measured effects reported in the paper

Surprising findings

• Baseline gains from the spatio-temporal aggregator and event-language adapter are small on Event-Chat (e.g., DC 3.40 to 3.52), though the paper frames them as significant.

Limitations

• Evaluation relies on GPT-based 1-5 scoring, with no human evaluation reported.
• Ablation gains are small and the paper does not report variance.
• Downstream detection and segmentation are only qualitative and rely on external heads (GroundingDINO, GroundedSAM).

Experimental setup

Datasets: LLaVA-Pretrain N-ImageNet DSEC E2VID N-ImageNet-Chat Event-Chat

Benchmarks: N-ImageNet-Chat test set Event-Chat test set

Metrics: Detail Captioning score (1-5) Complex Reasoning score (1-5) Visual Question Answering score (1-5)

Baselines: LLaVA-7B-v1.5 LLaVA-13B-v1.5 Qwen2-VL-7B InternVL2-8B DeepSeek-VL-7B EventCLIP EventBind

Builds on: LLaVA OpenCLIP Vicuna N-ImageNet GroundingDINO

Closest papers by meaning

• RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.91)
• Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.90)
• OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies (CVPR 2024 · similarity 0.89)
• E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
• ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.87)