Research map / Video / Video-language understanding
LLaFEA: Frame-Event Complementary Fusion for Fine-Grained Spatiotemporal Understanding in LMMs
Hanyu Zhou, Gim Hee Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add event camera input to frame input for LMMs (ablation) | LLaFEA · RealEvent-ST | t-IoU: 22.05 -> 52.80 | Improved |
| add frame input to event-only input (ablation) | LLaFEA · RealEvent-ST | s-IoU: 35.75 -> 46.27 | Improved |
| add event-based frame interpolation (w/ FI) to frame-only LMM baselines | VTimeLLM · RealEvent-ST | t-IoU: 21.05 -> 23.55 | Improved |
| add coordinate embedding of textual position and duration (ablation) | LLaFEA · RealEvent-ST | s-IoU: 14.15 -> 46.27 | Improved |
| add coordinate embedding (ablation) | LLaFEA · RealEvent-ST | DC: 3.62 -> 3.65 | No clear effect |
| add cross-attention spatial fusion only (ablation) | LLaFEA · RealEvent-ST | s-IoU: 4.03 -> 15.40 | Improved |
| add cross-attention temporal fusion only (ablation) | LLaFEA · RealEvent-ST | t-IoU: 6.15 -> 22.74 | Improved |
| add self-attention matching (ablation) | LLaFEA · RealEvent-ST | t-IoU: 6.15 -> 21.02 | Improved |
| combine cross-attention fusion with self-attention matching (ablation) | LLaFEA · RealEvent-ST | t-IoU: 21.02 -> 52.80 | Improved |
| use attention fusion instead of concatenation (ablation) | LLaFEA · RealEvent-ST | t-IoU: 40.32 -> 52.80 | Improved |
| use attention fusion instead of weighting (ablation) | LLaFEA · RealEvent-ST | s-IoU: 36.85 -> 46.27 | Improved |
| use staged separate training instead of joint training (ablation) | LLaFEA · RealEvent-ST | s-IoU: 38.05 -> 46.27 | Improved |
Closest papers by meaning
- LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding (CVPR 2025 · similarity 0.89)
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences (NeurIPS 2026 · similarity 0.88)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.87)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.87)
- Improving LLM Video Understanding with 16 Frames Per Second (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
