Research map / Vision-language and multimodal / Multimodal LLMs
SVLTA: Benchmarking Vision-Language Temporal Alignment via Synthetic Video Situation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SVLTA, a synthetic benchmark built from VirtualHome simulations with commonsense activity graphs, controllable action durations/permutations, and inequality-constrained filtering, to evaluate vision-language temporal alignment of VidLLMs and grounding models.
Problem
Existing vision-language temporal alignment benchmarks have biased temporal distributions, noisy human annotations, and limited compositionality, so they cannot fairly diagnose temporal alignment ability.
Approach
Generate synthetic home-activity videos in the VirtualHome simulator from commonsense-constrained action chains, control action positions and durations (ADD, AP), filter globally with inequality-constrained filtering, use template sentences, and derive timestamps automatically from the program execution log; define TJSD to measure temporal bias.
Key results
- Best closed-source VidLLM GPT-4o reaches only 18.90 mIoU on SVLTA temporal QA, and open VidLLMs such as Videochat2 get 0.87 mIoU, indicating weak temporal alignment.
- Shuffling, a debiased grounding model, shows the lowest robustness change RC of 1.04, while DCM has the highest RC of 17.86.
- In temporal alignment adaptation to Charades-STA, QD-DETR achieves 22.32 mIoU, outperforming 2D-TAN (11.8 mIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | 2D-TAN · SVLTA | mIoU: 76.41 -> 66.66 | Hurt |
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | VSLNet · SVLTA | mIoU: 92.63 -> 79.16 | Hurt |
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | LGI · SVLTA | mIoU: 85.25 -> 72.67 | Hurt |
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | QD-DETR · SVLTA | mIoU: 93.05 -> 87.72 | Hurt |
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | DCM · SVLTA | mIoU: 74.85 -> 58.88 | Hurt |
| Evaluate on the low-bias rather than high-bias test set after training on high-bias data | Shuffling · SVLTA | mIoU: 81.62 -> 80.36 | Hurt |
| Use Shuffling rather than DCM | Temporal alignment models trained on high-bias SVLTA · SVLTA high-bias versus low-bias test sets | RC: 17.86 -> 1.04 | Helped |
| Use QD-DETR rather than 2D-TAN for temporal alignment adaptation | Models transferred to Charades-STA · Charades-STA | mIoU: 11.8 -> 22.32 | Helped |
| Use QD-DETR rather than VSLNet for temporal alignment adaptation | Models transferred to Charades-STA · Charades-STA | mIoU: 19.66 -> 22.32 | Helped |
| Use GPT-4o rather than VTimeLLM for temporal question answering | Video-language models · SVLTA | mIoU: 10.29 -> 18.90 | Helped |
Surprising findings
- Debiased model DCM, which uses causal inference, is less robust to distribution shift than biased baselines (RC 17.86).
- Video-LLaMA2 (12.33 mIoU) outperforms time-aware VidLLMs such as VTimeLLM, attributed to its temporal encoding and high-resolution input.
Limitations
- Synthetic VirtualHome videos may have a domain gap from real-world videos.
- Sentences are template-based, limiting linguistic diversity despite GPT-3.5 rewriting.
- Closed-source models evaluated only on a 2000-sample subset.
Experimental setup
Datasets: SVLTA Charades-STA
Benchmarks: SVLTA Charades-STA
Metrics: R@1 at IoU 0.1/0.3/0.5/0.7/0.9 mIoU RC (robustness change) TJSD
Baselines: Video-LLaMA2 Video-ChatGPT Video-LLaVA Videochat2 LLaVA-Video E.T.Chat TimeChat VTimeLLM Gemini 1.5 Pro GPT-4o GPT-4o-mini 2D-TAN
Builds on: VirtualHome Charades-STA TimeChat VTimeLLM QD-DETR
Closest papers by meaning
- The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.88)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models (ICML 2026 · similarity 0.88)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.