Research map / Large language models / Instruction and fine-tuning
LISA: Reasoning Segmentation via Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces reasoning segmentation (implicit-query masks), the ReasonSeg benchmark, and LISA, a multimodal LLM that emits a <SEG> token whose embedding is decoded into a mask via SAM, trained end-to-end with LoRA.
Problem
Existing perception and segmentation systems need explicit referring expressions or fixed categories and cannot reason over implicit, complex user queries to produce segmentation masks; multimodal LLMs output only text.
Approach
Expand the LLM vocabulary with a <SEG> token and decode its last-layer hidden embedding into a mask with a SAM-based decoder (embedding-as-mask), enabling end-to-end training of text and mask outputs and zero-shot reasoning segmentation from reasoning-free data.
Key results
- LISA-7B achieves 36.8 gIoU / 34.1 cIoU on the ReasonSeg test set without reasoning training data, and LISA-13B-LLaVA1.5 (ft) reaches 61.3 gIoU / 62.2 cIoU on the val set.
- Fine-tuning on only 239 ReasonSeg samples boosts LISA-7B test gIoU from 36.8 to 47.3 (and LISA-13B to 51.7 gIoU).
- LISA-7B reaches 74.1 cIoU on refCOCO val, achieving state-of-the-art referring segmentation results among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LISA on 239 ReasonSeg samples | LISA-7B · ReasonSeg test | gIoU: 36.8 -> 47.3 | Helped |
| Use the LLaVA v1.5 base model | LISA-7B · ReasonSeg test | gIoU: 36.8 -> 48.7 | Helped |
| Use the LLaVA v1.5 13B base model and fine-tune on 239 ReasonSeg samples | LISA · ReasonSeg test | gIoU: 53.8 -> 61.3 | Helped |
| Use the embedding-as-mask end-to-end method instead of the two-stage LLaVA1.5 + OVSeg method | ReasonSeg test · ReasonSeg test | gIoU: 39.7 -> 48.7 | Helped |
| Use LISA instead of the strongest listed baseline | LISA-7B · ReasonSeg test | gIoU: 26.1 -> 36.8 | Helped |
| Apply LoRA to the SAM vision backbone (ablation) | LISA-7B · ReasonSeg val | gIoU: 44.4 -> 41.5 | Hurt |
| Use SAM pre-trained weights instead of training the vision backbone from scratch (ablation) | LISA-7B · ReasonSeg val | gIoU: 35.9 -> 52.9 | Helped |
| Add GPT-3.5 instruction rephrasing during fine-tuning (ablation) | LISA-7B · ReasonSeg val | gIoU: 50.7 -> 52.9 | Helped |
| Add the ReasonSeg validation samples to fine-tuning data (ablation) | LISA-7B · ReasonSeg test | gIoU: 51.7 -> 54.0 | Helped |
| Increase reasoning segmentation fine-tuning data from 239 to 439 samples (ablation) | LISA-7B · ReasonSeg test | cIoU: 51.1 -> 54.9 | Helped |
Surprising findings
- Zero-shot reasoning segmentation emerges when training only on reasoning-free semantic and referring segmentation datasets.
- Removing semantic segmentation data causes a large drop (gIoU 48.9 to 30.4 in the ablation), suggesting its binary masks are key supervision.
Limitations
- Performance bottleneck appears to lie in query understanding; the 7B model lags the 13B model especially on long queries.
- Two-stage baselines rely on text as an intermediary and need GPT-3.5 summarization when outputs exceed input limits.
Experimental setup
Datasets: ADE20K COCO-Stuff PACO-LVIS PartImageNet PASCAL-Part refCLEF refCOCO refCOCO+ refCOCOg LLaVA-Instruct-150k LLaVA-v1.5-mix665k ReasonSeg
Benchmarks: ReasonSeg refCOCO refCOCO+ refCOCOg
Metrics: gIoU cIoU
Baselines: OVSeg GRES X-Decoder SEEM Grounded-SAM LLaVA1.5 + OVSeg VisionLLM MCN VLT CRIS LAVT ReLA
Builds on: LLaVA SAM LoRA Mask2Former
Closest papers by meaning
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.91)
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · similarity 0.90)
- CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation (ICLR 2026 · similarity 0.89)
- Fast Reasoning Segmentation for Images and Videos (CVPR 2026 · similarity 0.89)
- Discriminative Perception via Anchored Description for Reasoning Segmentation (CVPR 2026 · similarity 0.88)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.