Research map / Vision-language and multimodal / Multimodal LLMs
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, Jianwei Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with grounded visual chat data and connect LLM to a grounding model | LLaVA-G · Grounding-Bench (1000 images) | F1: 32.0 (CogVLM-Grounding) -> 37.1 | Improved |
| allow gradients from grounding model to flow into the language model (no detach) (ablation) | LLaVA-G · Grounding-Bench (30 images) | F1: 35.1 -> 43.2 | Improved |
| detach grounding model gradient to LLM (ablation) | LLaVA-G · Grounding-Bench chat score | chat score (All): 79.3 -> 81.2 | Improved |
| require grounded output in the response (ablation) | LLaVA-G · LLaVA-Bench (COCO) | chat score (All): 81.2 -> 79.7 | No clear effect |
| increase grounding-model queries from 50 to 300 (ablation) | LLaVA-G · RefCOCO REC | ACC: 86.71 -> 86.35 | No clear effect |
| use marks overlaid on image as grounding prompt instead of the grounding model (ablation) | LLaVA-G · Flickr30K phrase grounding (test) | top1 accuracy: 83.6 -> 77.9 | Hurt |
| use box prompts instead of click prompts (ablation) | LLaVA-G · COCO val2017 object classification | accuracy: 70.8 -> 71.5 | Improved |
| use mark size 30 and mask alpha 0.2/0.2 instead of size 20 and alpha 0.4/0.4 (ablation) | LLaVA-G marks · Flickr30K phrase grounding (val) | top1 accuracy: 72.1 -> 76.6 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 0 of 1 on most shared results.
- MDETR — MDETR - Modulated Detection for End-to-End Multi-Modal Understanding (ICCV 2021) · better on 0 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.91)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.88)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.88)
- NExT-Chat: An LMM for Chat, Detection and Segmentation (ICML 2024 · similarity 0.87)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
