Research map / Detection and segmentation / Open-set and few-shot detection
NExT-Chat: An LMM for Chat, Detection and Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pix2emb, where an LMM emits a trigger token whose hidden state is decoded into boxes and SAM masks, and builds NExT-Chat, an LLaVA-style LMM for chat, detection, and segmentation.
Problem
Existing LMMs represent region coordinates as text tokens (pix2seq), which is inefficient for box output and cannot naturally produce fine-grained outputs such as segmentation masks.
Approach
Model locations as embeddings instead of text tokens: a <trigger> token's hidden state is decoded by a box MLP (L1+GIoU loss) or a SAM-based mask head, and location inputs are encoded into a single <loc> embedding via a location encoder trained with a cycle loss.
Key results
- NExT-Chat reaches 87.7 accuracy on POPE-Random vs. Shikra's 86.9.
- On RES, NExT-Chat averages 68.9 cIoU vs. LISA-7B's 67.9 and beats LISA on six of nine splits.
- On RefCOCOg region captioning, NExT-Chat achieves 79.6 CIDEr vs. Kosmos-2's 62.3 (4-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NExT-Chat instead of Shikra | POPE Random · POPE Random | accuracy: 86.90 -> 87.70 | Helped |
| Use NExT-Chat instead of Shikra | POPE Popular · POPE Popular | accuracy: 83.97 -> 84.57 | Helped |
| Use NExT-Chat instead of Shikra | POPE Adversarial · POPE Adversarial | accuracy: 83.10 -> 81.93 | Hurt |
| Use NExT-Chat instead of LISA-7B | RefCOCO RES · RefCOCO val and testA; testB | cIoU: 74.1 -> 74.7; 76.5 -> 78.9; 71.1 -> 69.5 | Mixed |
| Use NExT-Chat instead of LISA-7B | RefCOCO+ RES · RefCOCO+ val, testA and testB | cIoU: 62.4 -> 65.1; 67.4 -> 71.9; 56.5 -> 56.7 | Helped |
| Use NExT-Chat instead of LISA-7B | RefCOCOg RES · RefCOCOg val and test | cIoU: 66.4 -> 67.0; 68.5 -> 67.0 | Mixed |
| Use NExT-Chat instead of Shikra-13B | Referring expression detection · RefCOCO, RefCOCO+ and RefCOCOg | Acc@0.5: 85.5 vs 87.8; 90.0 vs 91.1; 77.9 vs 81.8; 77.2 vs 82.9; 84.5 vs 87.8; 68.0 vs 74.4; 80.1 vs 82.6; 79.8 vs 83.2 | Hurt |
| Use NExT-Chat instead of Kosmos-2 (4-shot) | RefCOCOg region captioning · RefCOCOg | CIDEr: 62.3 -> 79.6 | Helped |
| Use NExT-Chat instead of GRIT | RefCOCOg region captioning · RefCOCOg | METEOR: 15.2 -> 12.0 | Hurt |
Surprising findings
- NExT-Chat trails Shikra-7B on REC despite similar detection data, which the authors attribute to LM-loss vs. localization-loss balance and LLM not being pretrained on regression.
Limitations
- Trained on single-image inputs, so it cannot handle multiple images.
- Lacks diverse-domain training data, so performance on medical and satellite imagery is weak.
- Regression-based box output trails pix2seq (Shikra) on some REC splits.
Experimental setup
Datasets: Flickr30K Entities Visual Genome RefCOCO RefCOCO+ RefCOCOg VQAv2 PointQA Visual7W VCR LLaVA-instruct Shikra-RD
Benchmarks: POPE RefCOCO RefCOCO+ RefCOCOg RefCOCOg region captioning
Metrics: accuracy precision recall F1 cIoU Acc@0.5 CIDEr METEOR
Baselines: Shikra LISA-7B Kosmos-2 InstructBLIP MiniGPT-4 LLaVA MM-GPT mPLUG-Owl GRIT ASM VisionLLM-H UniTAB
Builds on: LLaVA pix2seq SAM LISA Shikra
Closest papers by meaning
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.89)
- GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · similarity 0.87)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.87)
- Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.87)
- Ferret: Refer and Ground Anything Anywhere at Any Granularity (ICLR 2024 · similarity 0.86)
- Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.