Research map / Detection and segmentation / Open-set and few-shot detection
See Say and Segment: Teaching LMMs to Overcome False Premises
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FP-RefCOCO(+/g) false-premise referring segmentation benchmark and SESAME, an LMM (LLaVA-v1.5-7B + SAM with LoRA) jointly trained to see, say, and segment without catastrophic forgetting.
Problem
Open-source LMM-based reasoning/referring segmentation models hallucinate masks for queries about objects absent from the image and lose the ability to detect absence and respond naturally (catastrophic forgetting after segmentation fine-tuning).
Approach
Augment RefCOCO(+/g) with LLM-generated context-aware false-premise queries paired with corrections, then jointly train a single LLaVA-based LMM with SAM decoder on a mixed set (FP-RefCOCO, LLaVA VQA, R-RefCOCO) so it can detect, reject/correct, and segment within one model; also a cascading two-LMM baseline.
Key results
- SESAME reaches 79.84% see accuracy on FP-RefCOCO versus 51.36% for LISA-7B, and 81.78% on FP-RefCOCOg.
- On FP-RefCOCOg, SESAME achieves 53.79 cIoU versus 39.64 for LISA-7B, with its advantage holding at 75% false-premise sampling.
- On standard positive-only referring segmentation, SESAME scores 74.7/64.9/66.1 on refCOCO/refCOCO+/refCOCOg, close to LISA-7B's 74.9/65.1/67.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCO training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCO val | See accuracy: 65.11 -> 79.84 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCO training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCO val | Say (CLAIR): 0.00 -> 0.63 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCO training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCO val | Segment (cIoU): 47.47 -> 57.93 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCO+ training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCO+ val | See accuracy: 64.00 -> 80.00 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCO+ training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCO+ val | Segment (cIoU): 37.61 -> 50.81 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCOg training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCOg val | See accuracy: 58.15 -> 81.78 | Helped |
| Fine-tune SESAME jointly for seeing, saying, and segmentation on FP-RefCOCOg training data | LLaVA-v1.5-7B with SAM decoder · FP-RefCOCOg val | Segment (cIoU): 35.15 -> 53.79 | Helped |
| Jointly fine-tune for false-premise robustness and segmentation rather than using LISA-7B | LLaVA-v1.5-7B with SAM decoder · refCOCO | segmentation: 74.9 -> 74.7 | No clear effect |
| Jointly fine-tune for false-premise robustness and segmentation rather than using LISA-7B | LLaVA-v1.5-7B with SAM decoder · refCOCO+ | segmentation: 65.1 -> 64.9 | No clear effect |
| Jointly fine-tune for false-premise robustness and segmentation rather than using LISA-7B | LLaVA-v1.5-7B with SAM decoder · refCOCOg | segmentation: 67.9 -> 66.1 | No clear effect |
| Increase false-premise query proportion from 0% to 75% at test time (ablation) | SESAME · FP-RefCOCOg | Segment (cIoU): 66.02 -> 49.79 | Hurt |
| Add R-RefCOCO(+/g) to unified training so the model can reject false premises without proposing alternatives (ablation) | SESAME · FP-RefCOCO(+/g) | Say capability | Helped |
Surprising findings
- LISA loses its see and say abilities entirely (0% recall on false premises, CLAIR near 0) despite its base LLaVA model having them, indicating catastrophic forgetting from segmentation fine-tuning.
- Training only on LLM-generated corrections causes the model to produce speculative edits such as flipping 'left' to 'right' and hallucinating the reversed object, so rejection-only data (R-RefCOCO) is needed.
Limitations
- Highest false-premise recall is only 67.89%, so detection remains considerably imperfect.
- Say scores are computed with a GPT-4-based CLAIR metric, which depends on an external closed judge.
- Relies on GPT-3.5-turbo generated negatives, which may contain errors; data cleaning only ensured parseability.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg LLaVA VQA instruction finetuning data R-RefCOCO
Benchmarks: FP-RefCOCO FP-RefCOCO+ FP-RefCOCOg RefCOCO RefCOCO+ RefCOCOg
Metrics: binary classification accuracy (see) CLAIR score (say) cIoU (segment)
Baselines: LISA-7B GRES SEEM Cascading (LLaVA-v1.5 + LISA) MCN VLT CRIS LAVT ReLA X-Decoder
Builds on: LISA LLaVA-v1.5 Segment Anything RefCOCO
Closest papers by meaning
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.88)
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.88)
- Seeing What Matters: A Training-Free Self-Guided Framework for Multimodal Detail Perception and Reasoning (CVPR 2026 · similarity 0.88)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.