Research map / Video / Video-language understanding
Spatio-Temporal LLM: Reasoning about Environments and Actions
Haozhen Zheng, Beitong Tian, Mingyuan Wu, Zhenggang Tang, Klara Nahrstedt, Alex Schwing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune on REA dataset | LLaVA-Video-7B-Qwen2 · REA test | LLM-judge overall (ChatGPT-4o): 30.96 -> 36.99 | Improved |
| finetune on REA dataset | LLaVA-OV-Qwen2-7B · REA test | LLM-judge overall (ChatGPT-4o): 23.85 -> 38.19 | Improved |
| add spatial point-cloud queries via STLLM-Aligner to finetuned LLaVA-Video | LLaVA-Video-7B-Qwen2 · REA test | LLM-judge overall (ChatGPT-4o): 36.99 -> 41.89 | Improved |
| add direct 3D feature concatenation (STLLM-3D) to finetuned LLaVA-Video | LLaVA-Video-7B-Qwen2 · REA test | LLM-judge overall (Gemini 2.0 Flash): 42.92 -> 46.39 | Improved |
| finetune LLM layers in STLLM-Aligner (not frozen) (ablation) | STLLM-Aligner (w Pos. Enc.) · REA test | LLM-judge overall (ChatGPT-4o): 35.38 -> 41.43 | Improved |
| finetune LLM layers in STLLM-Aligner (not frozen) (ablation) | STLLM-Aligner (w Pos. Enc.) · REA test | CIDEr: 170.63 -> 406.54 | Improved |
| add high-frequency positional encoding to STLLM-Aligner (ablation) | STLLM-Aligner · REA test | LLM-judge overall (ChatGPT-4o): 41.89 -> 41.43 | No clear effect |
| add high-frequency positional encoding to STLLM-Aligner (ablation) | STLLM-Aligner · SQA3D test (zero-shot) | average correct rate (GPT judge): 54.40 -> 53.32 | Hurt |
| finetune on REA dataset | LLaVA-Video-7B-Qwen2 · SQA3D test (zero-shot) | average correct rate (GPT judge): 43.04 -> 52.03 | Improved |
| finetune on REA dataset | LLaVA-OV-Qwen2-7B · SQA3D test (zero-shot) | average correct rate (GPT judge): 43.98 -> 44.29 | No clear effect |
| use STLLM-3D on top of REA-finetuned LLaVA-Video | LLaVA-Video-7B-Qwen2 · SQA3D test (zero-shot) | average correct rate (GPT judge): 52.03 -> 55.21 | Improved |
Closest papers by meaning
- OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding (NeurIPS 2025 · similarity 0.91)
- Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.90)
- Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.90)
- From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.90)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
