Research map / Vision-language and multimodal / Multimodal LLMs

Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Event-MLLM, which fuses event streams with degraded RGB frames via a DINOv2-derived Illumination Indicator and an Illumination Correction feature-alignment loss, and releases a multi-illumination event-instruction dataset and benchmark for MLLMs under extreme lighting.

Problem

MLLMs fail under extreme illumination (overexposure or darkness) where RGB inputs lose structure and semantics; event cameras could help but have not been integrated into MLLMs for reasoning/instruction following.

Approach

A learnable Illumination Indicator from a frozen DINOv2 branch modulates event-RGB fusion, and an Illumination Correction loss (MSE) aligns fused features to normal-light features in the vision encoder space, so inference needs only degraded RGB plus events.

Key results

• Ours-7B reaches 53.13% multi-choice accuracy and 85.43% F1 vs. 44.71% and 81.95% for Baseline-7B, averaged over 17 brightness levels.
• Ours-7B object-counting accuracy 74.66% and MAE 0.4557, improving on Baseline-7B (72.73%, 0.5303) and InternVL3 (72.56%).
• Illumination Correction raises cosine similarity of underexposed features (e.g., Qwen-3B at 0.05 brightness) from 0.66 to 0.93.

Surprising findings

• EventGPT (event-only) gets only 3.16% multi-choice accuracy despite a 73.60% F1, a mismatch between accuracy and F1 metrics.

Limitations

• Relies on event-camera data, which may be sparse or unavailable in some settings.
• Dataset covers limited real-world conditions and instruction types.

Experimental setup

Datasets: four spatially and temporally matched event-RGB paired datasets

Benchmarks: multi-illumination event-instruction benchmark (Multiple-Choice and Object-Counting)

Metrics: Accuracy F1 score MAE cosine similarity

Baselines: LLaVA-7B DeepSeek-VL InternVL3 EventGPT Q-Instruct Baseline-3B Baseline-7B

Builds on: LoRA DINOv2 Qwen2.5-VL

Closest papers by meaning

• RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.91)
• EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
• Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention (CVPR 2026 · similarity 0.89)
• POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs (CVPR 2026 · similarity 0.88)
• Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.88)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.88)