Research map / Vision-language and multimodal / Multimodal LLMs
LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaVA-ST, an MLLM that unifies spatial, temporal, and spatial-temporal interleaved fine-grained video/image understanding via language-aligned positional embeddings, a spatial-temporal packer, and a three-stage training on the 4.3M-sample ST-Align dataset.
Problem
Existing MLLMs handle either spatial or temporal fine-grained localization but not both jointly; joint spatial-temporal coordinate alignment and compressing video features while preserving fine-grained detail are difficult.
Approach
Embeds coordinate special-token embeddings (from both input embedding and LM output head) as positional embeddings added to visual features (LAPE), and compresses video features with separate point-to-region attention packers for spatial and temporal dimensions (STP), trained progressively on coarse-to-fine data.
Key results
- On Charades-STA TVG, LLaVA-ST reaches 42.4 mIoU vs 36.8 for Grounded-VideoLLM.
- On the ST-Align STVG task, LLaVA-ST gets 43.8 m_t IoU and 22.8 m_s IoU, versus 33.0 and 9.2 for the best baselines reported there.
- On MVBench, LLaVA-ST averages 64.2 vs 59.4 for Grounded-VideoLLM, the next best reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaVA-ST instead of Grounded-VideoLLM | STVG · ST-Align | m_tIoU: 33.0 -> 43.8 | Helped |
| Use LLaVA-ST instead of GroundingGPT | STVG · ST-Align | m_sIoU: 9.2 -> 22.8 | Helped |
| Use LLaVA-ST instead of Grounded-VideoLLM | Charades-STA temporal grounding · Charades-STA | mIoU: 36.8 -> 42.4 | Helped |
| Use LLaVA-ST instead of Groma-7B | RefCOCO+ validation · RefCOCO+ val | accuracy: +2.1% | Helped |
| Use LLaVA-ST instead of previous models | MVBench · MVBench | Avg.: +4.8 | Helped |
| Add LAPE (ablation) | LLaVA-ST · RefCOCO val | Acc@0.5: 88.7 -> 89.6 | Helped |
| Add LAPE (ablation) | LLaVA-ST · ST-Align STVG | m_tIoU: 42.1 -> 43.5 | Helped |
| Add LAPE (ablation) | LLaVA-ST · ST-Align STVG | m_sIoU: 17.9 -> 21.0 | Helped |
| Add LAPE (ablation) | LLaVA-ST · Charades-STA | mIoU: 36.1 -> 38.5 | Helped |
| Add STP instead of direct pooling (ablation) | LLaVA-ST · Charades-STA | mIoU: 35.5 -> 38.5 | Helped |
| Add spatial data to temporal-only training data (ablation) | LLaVA-ST · Charades-STA | mIoU: 34.6 -> 32.3 | Hurt |
| Add spatial-temporal interleaved training data (ablation) | LLaVA-ST · Charades-STA | mIoU: 34.6 -> 38.5 | Helped |
Surprising findings
- Adding spatial-only data to temporal-only training decreased Charades-STA mIoU by 2.3, while adding spatial-temporal interleaved data improved it.
Limitations
- Baselines for ST-Align interleaved tasks were constructed by the authors via multi-step prompting pipelines, which may not reflect the best possible baselines.
- Spatial-temporal interleaved evaluation relies on a self-collected benchmark built from GPT-4-turbo revised VidSTG annotations.
Experimental setup
Datasets: ST-Align WebVid-10M Panda-70M InternVid-10M VTimeLLM-Stage2 Moment-10M InternVid-G GranD VidSTG Charades-STA RefCOCO RefCOCO+
Benchmarks: ST-Align benchmark (STVG, ELC, SVG) Charades-STA RefCOCO RefCOCO+ RefCOCOg MSVD-QA MSRVTT-QA VCG-Bench MVBench
Metrics: tIoU@0.5 mIoU sIoU@0.5 mIoU (temporal/spatial) METEOR Acc@0.5 R@0.3 R@0.5 accuracy VCG score
Baselines: GroundingGPT VTimeLLM Grounded-VideoLLM Video-LLaMA SeViLA Video-ChatGPT VideoChat2 Groma Ferret VideoGPT+ ST-LLM MVBench leaderboard models
Builds on: LLaVA-OneVision SigLIP VTimeLLM Grounded-VideoLLM GroundingGPT
Closest papers by meaning
- TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations (AAAI 2025 · similarity 0.90)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.89)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.88)
- Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.88)
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.