Research map / Vision-language and multimodal / Multimodal LLMs
DanmakuTPPBench: A Multi-modal Benchmark for Temporal Point Process Modeling and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DanmakuTPPBench, a multi-modal temporal point process benchmark built from Bilibili danmaku (DanmakuTPP-Events) plus a multi-agent-generated QA set (DanmakuTPP-QA), evaluating classical neural TPP models and LLMs/MLLMs.
Problem
Existing TPP datasets are unimodal and lack textual and visual context, so there is no benchmark for joint temporal, textual, and visual reasoning over event sequences with LLMs/MLLMs.
Approach
Uses timestamped Bilibili danmaku comments as natively multi-modal TPP events (time, text, video frame), and builds a 10-task QA benchmark via a multi-agent pipeline where LLMs/MLLMs design tasks, annotate, quality-control, visualize, and solve them.
Key results
- On DanmakuTPP-Events, NHP achieves the best log-likelihood (0.799) and near-best next-event RMSE (0.932), with THP at 0.938.
- On DanmakuTPP-QA closed-ended tasks, finetuned Qwen2.5-VL-3B reaches RMSE 0.05, 0.16, 0.08 on sentiment tasks T-4 to T-6, a 4-6x error reduction over the best pretrained models.
- Qwen3-235B-A22B scores 0.52 on open-ended Task-10, the highest among evaluated models; Qwen2.5-VL-72B scores 0.48 on Task-9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale Qwen2.5-Instruct from 7B to 32B | Qwen2.5-Instruct · DanmakuTPP-QA Task-2 | RMSE: 27.64 to 1.52 | Helped |
| Scale Qwen2.5-Instruct from 7B to 72B | Qwen2.5-Instruct · DanmakuTPP-QA Task-2 | RMSE: 27.64 to 1.28 | Helped |
| Use an MLLM instead of an LLM | DanmakuTPP-QA · Tasks 1–8 | task-specific accuracy or RMSE | Mixed |
| LoRA fine-tune Qwen2.5-VL-3B on DanmakuTPP-QA | Qwen2.5-VL-3B · DanmakuTPP-QA Tasks 4–6 | RMSE: 0.05, 0.16, and 0.08 | Helped |
| LoRA fine-tune Qwen2.5-VL-3B on DanmakuTPP-QA | Qwen2.5-VL-3B · DanmakuTPP-QA Task-3 | RMSE: 220.43 | Hurt |
| LoRA fine-tune Qwen2.5-VL-3B on DanmakuTPP-QA | Qwen2.5-VL-3B · DanmakuTPP-QA Tasks 9–10 | LLM-Eval correctness: 0.36 and 0.30 | Helped |
Surprising findings
- MLLMs given visual frames do not consistently outperform text-only LLMs on TPP QA tasks.
- Finetuning Qwen2.5-VL-3B degrades Task-3 RMSE to 220.43, worse than all pretrained models.
Limitations
- Dataset is primarily Chinese-language content, limiting generalizability across languages and cultures.
- Annotation quality depends on underlying models, which may introduce biases or inconsistencies.
- Visual input limited to three sampled frames per time window due to compute constraints.
Experimental setup
Datasets: DanmakuTPP-Events DanmakuTPP-QA Retweet StackOverflow Taobao Taxi ActiveRT RNCNIX Amazon Review
Benchmarks: DanmakuTPP-Events DanmakuTPP-QA
Metrics: RMSE accuracy log-likelihood correctness score (0-1)
Baselines: Neural Hawkes Process RMTPP SAHP THP AttNHP ODETPP Qwen2.5-7B-Instruct Qwen3-235B-A22B DeepSeek-V3 Qwen2.5-VL-72B Gemma3-27B
Builds on: EasyTPP LoRA
Closest papers by meaning
- EasyTPP: Towards Open Benchmarking Temporal Point Processes (ICLR 2024 · similarity 0.89)
- TimE: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios (NeurIPS 2025 · similarity 0.87)
- TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.87)
- Episodic Memories Generation and Evaluation Benchmark for Large Language Models (ICLR 2025 · similarity 0.87)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- EventLens: Event-Structure Reinforcement Learning for Video Understanding (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.