Research map / Video / Video-language understanding
SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAMA-239K dataset, SAMA video LMM with a spatio-temporal context aggregator plus SAM2 decoder, and SAMA-Bench for multi-turn referential grounded video chat.
Problem
Video LMMs handle video referring understanding and video grounding separately; there is no unified, high-quality instruction data or benchmark for multi-turn referentially grounded video chat.
Approach
Build a Gemini-1.5 Pro-driven pipeline that turns referring-segmentation and grounding videos into multi-turn grounded QA with object descriptions, and extend Sa2VA with a three-stage spatial-temporal-context aggregator feeding a SAM2 decoder so the model can both describe and segment referred objects in video.
Key results
- SAMA-8B achieves 0.70 mIoU and 0.55 Recall on SAMA-Bench-G, exceeding Ferret-13B+SAM2 by 6% mIoU and 12% Recall.
- SAMA-8B reaches 53.7 on MeViS, 77.3 on Ref-DAVIS17, and 59.1 on ReVOS, outperforming Sa2VA-8B by 2.2, 1.4, and 1.5 points.
- Removing SAMA-239K drops SAMA-4B Recall from 0.55 to 0.10 and CIDEr from 0.65 to 0.02 on SAMA-Bench-G.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the spatial-temporal-context aggregator to SAMA-4B (ablation) | SAMA-4B · SAMA-Bench-G | mIoU: 0.66 -> 0.69 | Helped |
| Add the spatial-temporal-context aggregator to SAMA-4B (ablation) | SAMA-4B · SAMA-Bench-G | CLAIR: 0.50 -> 0.57 | Helped |
| Add the SAMA-239K dataset to training (ablation) | SAMA-4B · SAMA-Bench-G | Recall: 0.10 -> 0.55 | Helped |
| Add the SAMA-239K dataset to training (ablation) | SAMA-4B · SAMA-Bench-G | CIDEr: 0.02 -> 0.65 | Helped |
| Add the spatial-temporal-context aggregator (ablation) | SAMA-4B · MeViS (val_u) | J&F: 55.4 -> 57.9 | Hurt |
| Add the SAMA-239K dataset to training (ablation) | SAMA-4B · Ref-DAVIS | J&F: 74.2 -> 74.0 | No clear effect |
| Increase LLaVA-v1.5-mix665k training data from 50% to 100% (ablation) | SAMA · Image chat benchmarks | MMBench: 43.6 -> 55.8 | Helped |
| Increase next-token prediction loss weight from 1.00 to 1.50 (ablation) | SAMA · Chat benchmarks | AI2D: 57.8 -> 60.0 | Helped |
| Increase next-token prediction loss weight from 1.00 to 1.25 (ablation) | SAMA · SAMA-Bench-G | mIoU: 67.1 -> 66.4 | Hurt |
| Use a box prompt instead of one point (ablation) | SAMA · SAMA-Bench-G | mIoU: 0.60 -> 0.66 | Helped |
| Use a mask prompt instead of a box prompt (ablation) | SAMA · SAMA-Bench-G | Recall: 0.49 -> 0.50 | Helped |
| Use an 8-point prompt instead of a 1-point prompt (ablation) | SAMA · SAMA-Bench-G | mIoU: 0.60 -> 0.64 | Helped |
Surprising findings
- Removing the SAMA-239K dataset nearly collapses grounded dialogue performance (Recall 0.10, CIDEr 0.02).
- SAMA slightly trails Sa2VA on general image and video chat benchmarks (e.g., MMBench 80.8 vs 84.4, Video-MME 51.8 vs 54.3), attributed to joint-training interference.
Limitations
- Only box-based prompts are supported natively; points and masks are not handled in the main model.
- Lags specialized conversational models on general dialogue tasks, and joint grounding and dialogue training causes interference.
- SAMA-239K is limited to short clips rather than long videos.
Experimental setup
Datasets: MeViS Ref-YouTube-VOS LV-VIS SAV VidSTG LLaVA-1.5 665K ChatUniVi refCOCO refCOCO+ refCOCOg GRand-F ReVOS
Benchmarks: SAMA-Bench MeViS Ref-DAVIS17 Ref-YTVOS ReVOS RefCOCO RefCOCO+ RefCOCOg GCG MME MMBench SEED-Bench
Metrics: mIoU Recall METEOR CIDEr CLAIR cIoU J&F accuracy
Baselines: Sa2VA-1B Sa2VA-4B Sa2VA-8B Ferret-7B Ferret-13B GLaMM Shikra VISA-13B VideoLISA-3.8B VideoGLaMM Gemini-1.5 Pro Gemini-2.0 Flash
Builds on: Sa2VA SAM2 Q-Former HQ-SAM GLaMM
Closest papers by meaning
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.91)
- Artemis: Towards Referential Understanding in Complex Videos (NeurIPS 2024 · similarity 0.89)
- SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding (ICLR 2025 · similarity 0.89)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.