Research map / Vision-language and multimodal / Multimodal LLMs
INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an instance-grounded image and video instruction-tuning dataset (Inst-IT) with set-of-marks visual prompts and GPT-4o annotations, a human-verified instance understanding benchmark, and a continuous SFT recipe for LLaVA-NeXT.
Problem
Large multimodal models struggle with fine-grained instance-level understanding in both images and videos, and existing instruction-tuning data and benchmarks lack instance-level annotations.
Approach
Annotate images and videos with set-of-marks visual prompts (numeric IDs on instances, using ground-truth or SAM-derived masks) via GPT-4o at instance, frame, temporal-difference, and video levels, then fine-tune LLaVA-NeXT in a continuous SFT stage with partially frozen vision encoder to keep general ability.
Key results
- LLaVA-NeXT-Inst-IT (Vicuna-7B) improves over LLaVA-NeXT by 4.4 points on AI2D and 13.5 on ChartQA, and by 7.8 on EgoSchema and 11.8 on NExT-QA.
- With Qwen2-7B and SigLIP, the model reaches 78.7 on AI2D and 72.8 on ChartQA, and 54.0 on VideoMME, close to or above LLaVA-OV and Qwen2-VL-Instruct on several benchmarks.
- On RefCOCOg, the Vicuna-7B Inst-IT model improves from 52.2% to 63.0%, a 10.8% gain over the LLaVA-NeXT baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · AI2D test | accuracy: 66.6 -> 71.0 | Helped |
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · ChartQA test | accuracy: 54.8 -> 68.3 | Helped |
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · EgoSchema subset | accuracy: 50.0$^*$ -> 57.8 | Helped |
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · NExTQA MC | accuracy: 58.4$^*$ -> 70.2 | Helped |
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · ANetQA OE | score: 53.8 -> 53.7 | Hurt |
| Train with Inst-IT instead of the baseline | LLaVA-NeXT, Vicuna-7B · RefCOCOg | accuracy: 52.2% -> 63.0% | Helped |
| Add Inst-IT video data to LLaVA-NeXT-Data (ablation) | LLaVA-NeXT continuous SFT recipe · Inst-IT video MC | accuracy: 31.3 -> 43.0 | Helped |
| Add Inst-IT video data to LLaVA-NeXT-Data (ablation) | LLaVA-NeXT continuous SFT recipe · AI2D test | accuracy: 61.1 -> 60.7 | Hurt |
| Use continuous SFT and freeze the first 12 vision-encoder layers (ablation) | LLaVA-NeXT with video Inst-IT data · AI2D test | accuracy: 60.7 -> 63.2 | Helped |
| Add Inst-IT image data to video Inst-IT data (ablation) | LLaVA-NeXT with continuous SFT · Inst-IT image MC | accuracy: 60.2 -> 58.6 | Hurt |
| Add instance-level and image-level captions (ablation) | LLaVA-NeXT · Inst-IT image MC | accuracy: 45.3 -> 58.9 | Helped |
| Add temporal-difference annotations (ablation) | LLaVA-NeXT with instance and image captions · Inst-IT video MC | accuracy: 33.8 -> 36.9 | Helped |
Surprising findings
- Directly mixing video instance data with generic SFT data degrades generic image understanding, which the authors address by a continuous SFT stage with frozen early vision encoder layers.
- Scaling Qwen2-VL from 7B to 72B does not substantially improve instance understanding on Inst-IT-Bench, and large coarse-grained data volumes do not help either.
Limitations
- Experiments limited to 7B and 1.5B models due to compute cost.
- Data pipeline depends on GPT-4o API and is constrained by its cost.
- Failure cases on heavily occluded, blurry, tiny, or crowded instances.
Experimental setup
Datasets: Inst-IT Dataset LLaVA-NeXT-DATA LCS-558K BRUST UVO OVIS LVVIS YoutubeVIS-2021 BenSMOT VidOR SA-1B
Benchmarks: Inst-IT Bench ViP-Bench RefCOCOg AI2D MMMU POPE GQA ChartQA ANetQA EgoSchema NExT-QA VideoMME
Metrics: accuracy GPT-4o-judged similarity score F1
Baselines: LLaVA-NeXT LLaVA-1.5 ViP-LLaVA SoM-LLaVA LLaVA-OV LLaVA-Video Qwen2-VL-Instruct GPT-4o
Builds on: LLaVA-NeXT set-of-marks prompting SoM-LLaVA ViP-LLaVA
Closest papers by meaning
- Learning to Instruct for Visual Instruction Tuning (NeurIPS 2025 · similarity 0.89)
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.88)
- LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.88)
- Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners (ICML 2026 · similarity 0.87)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.87)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.