Research map / Video / Video-language understanding
TRACE: Temporal Grounding Video LLM via Causal Event Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces causal event modeling, which represents video LLM outputs as sequences of (timestamp, salient score, caption) events, and TRACE, a task-interleaved video LLM with separate encoders/heads per task on Mistral-7B that improves zero-shot and fine-tuned video temporal grounding.
Problem
Video LLMs for video temporal grounding (VTG) generate pure natural language, which does not model the structured event nature of videos (timestamps, salient scores, captions), limiting performance on VTG tasks.
Approach
Model video LLM outputs as causally ordered event triplets (timestamp, score, caption), predicting each event from prior events, video, and instructions; implement via task-interleaved tokens with separate encoders and decoding heads for time, score, and text, and adaptive head-switching triggered by a sync token.
Key results
- Zero-shot TRACE (7B) reaches 40.3 R@1 IOU=0.5 on Charades-STA vs 33.8 for VTG-LLM and 26.8 mAP on QVHighlights vs 16.5.
- After fine-tuning on Youcook2, TRACE reaches CIDEr 35.5 and F1 31.8, beating CM^2 (31.7 CIDEr, 28.4 F1).
- On ActivityNet Captions, TRACE attains 37.7 R@1 IOU=0.5 and 39.0 mIOU, best among compared video LLMs for moment retrieval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use causal event modeling instead of natural-language-style outputs (ablation) | TRACE · Youcook2 | CIDEr: 4.3 -> 6.9 | Helped |
| Use independent task encoders and decoding heads instead of shared text tokenization/heads (ablation) | TRACE · Youcook2 and Charades-STA | instruction following | Helped |
| Increase sampled frames from 8 to 64 (ablation) | TRACE (VTG-IT) · Youcook2 | CIDEr: 5.0 -> 6.9 | Helped |
| Increase sampled frames from 64 to 128 (ablation) | TRACE (VTG-IT) · Charades-STA | R@1 (IoU=0.5): 37.0 -> 41.2 | Helped |
| Add ActivityNet Captions and InternVid training data to VTG-IT (ablation) | TRACE · ActivityNet Captions | R@1 (IoU=0.5): 19.2 -> 37.7 | Helped |
| Add ActivityNet Captions and InternVid training data to VTG-IT (ablation) | TRACE · Charades-STA | R@1 (IoU=0.5): 41.2 -> 40.3 | Hurt |
| Add general video understanding data from LLaVA-Video-178k (ablation) | TRACE-uni · Charades-STA | R@1 (IoU=0.5): 40.3 -> 43.7 | Helped |
| Increase slots per frame from 8 to 16 (ablation) | TRACE (VTG-IT, 64 frames) · Charades-STA | R@1 (IoU=0.5): 37.0 -> 41.9 | Helped |
| Use TRACE instead of VTG-LLM | 7B video LLMs · QVHighlights zero-shot | mAP: 16.5 -> 26.8 | Helped |
Surprising findings
- Directly adding time and score tokens to the text tokenizer (shared heads) caused the model to fail to follow instructions, highlighting disruption of pretrained LLM knowledge.
Limitations
- Relies on decoder-only LLM conditioning only on previous events, so complex inter-event relationships are not modeled.
- Trails general video LLMs on MVBench and VideoMME (TRACE 48.1 and 43.8 vs VideoLLaMA2 54.6 and 46.6).
Experimental setup
Datasets: Valley LLaVA_Image TextVR ShareGPT4Video VTG-IT ActivityNet Captions InternVid VideoChatGPT Next-QA LLaVA-Video-178k HiREST COIN
Benchmarks: Youcook2 ActivityNet Captions Charades-STA QVHighlights MVBench VideoMME E.T.Bench Event-Bench
Metrics: SODA_c CIDEr METEOR F1 Score R@1 IOU=0.5 R@1 IOU=0.7 mIOU mAP HIT@1 accuracy
Baselines: Valley VideoChat Video-LLaMA TimeChat VTimeLLM Momentor HawkEye VTG-LLM Vid2Seq PDVC CM^2 Moment-DETR
Builds on: VTG-LLM Vid2Seq Slot-based compression Mistral-7B
Closest papers by meaning
- VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding (AAAI 2025 · similarity 0.93)
- Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025 · similarity 0.89)
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · similarity 0.87)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.87)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.86)
- TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.