Research map / Video / Video-language understanding
VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoGLaMM, a video large multimodal model with dual spatial/temporal encoders and a spatio-temporal SAM2-based pixel decoder, trained on a new 38k-triplet grounded video conversation dataset for pixel-level grounding in videos.
Problem
Existing video LMMs handle conversation but cannot produce pixel-level, spatio-temporally consistent grounding of their responses to objects in videos.
Approach
Couple an LLM (Phi-3-Mini with LoRA) with separate spatial (CLIP) and temporal (InternVideo2) encoders via tunable V→L adapters and an L→V adapter feeding a SAM2-initialized spatio-temporal pixel decoder, so the LLM's <SEG> outputs yield masks interleaved with text; train on a semi-automatically annotated grounded video conversation dataset.
Key results
- On GCG, VideoGLaMM reaches 62.34 mIoU vs 28.60 for GLaMM+SAM2 and 24.03 for PG-Video-LLaVA.
- On MeViS, VideoGLaMM reaches J&F 45.15 vs 44.40 for VideoLISA (post-processed).
- On VidSTG interrogative visual grounding, VideoGLaMM reaches 39.66 mIoU vs 38.63 for GLaMM+SAM2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VideoGLaMM instead of GLaMM + SAM2 | GCG generation · GCG test | mIoU: 28.60 -> 62.34 | Helped |
| Use VideoGLaMM instead of VideoLISA | referring video segmentation · MeViS validation | J &F: 44.40 -> 45.15 | Helped |
| Use VideoGLaMM instead of VideoLISA | referring video segmentation · Ref-DAVIS-17 | J &F: 68.8 -> 69.5 | Helped |
| Use VideoGLaMM instead of GLaMM + SAM2 | visual grounding · VidSTG interrogative | mIoU: 38.63 -> 39.66 | Helped |
| Use only the image encoder instead of the dual encoder (ablation) | VideoGLaMM · GCG test | mIoU: 62.34 -> 60.06 | Hurt |
| Use only the video encoder instead of the dual encoder (ablation) | VideoGLaMM · GCG test | CLAIR: 28.2 -> 26.5 | Hurt |
| Use only the video encoder instead of the dual encoder (ablation) | VideoGLaMM · GCG test | mIoU: 62.34 -> 64.62 | Helped |
| Remove the temporal branch from the pixel decoder (ablation) | VideoGLaMM · GCG test | mIoU: 62.34 -> 59.68 | Hurt |
| Use 4 rather than 8 input frames for the pixel decoder (ablation) | VideoGLaMM · GCG test | mIoU: 62.34 -> 63.82 | Helped |
| Use 4 rather than 8 input frames for the pixel decoder (ablation) | VideoGLaMM · GCG test | METEOR: 0.103 -> 0.094 | Hurt |
Surprising findings
- Using only the video encoder gives the highest GCG mIoU (64.62), higher than the dual encoder (62.34), though conversational metrics are lower.
Limitations
- Annotation noise remains despite manual validation; not all objects are densely described.
- Struggles with objects of varying granularities.
- Designed for short-medium duration clips; not extended to long videos.
Experimental setup
Datasets: YTVIS BURST ActivityNet entities Refer-YTVOS MeViS VidSTG HCSTVG ADE20K COCO-Stuff LVIS-PACO refCOCO refCOCO+
Benchmarks: GCG evaluation set (308 video-QA triplets) VidSTG MeViS Ref-DAVIS-17
Metrics: mIoU Recall METEOR CIDEr CLAIR J F J&F
Baselines: PG-Video-LLaVA GLaMM + SAM2 VideoLISA LMPM LISA TrackGPT
Builds on: GLaMM SAM2 PG-Video-LLaVA VideoChat-GPT
Closest papers by meaning
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.92)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.91)
- PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? (NeurIPS 2026 · similarity 0.89)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.89)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · similarity 0.89)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.