Research map / Vision-language and multimodal / Multimodal LLMs

Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FCLM, an MLLM that unifies region-level captioning and pixel-level grounding via a consistency loss aligning visual-prompt <pos> tokens with [SEG] tokens, plus a hybrid region extractor and a new DL-RES task.

Problem

Existing MLLMs for fine-grained visual understanding are mostly task-specific, and unified approaches do not explicitly exploit the correspondence between visual prompt embeddings (for captioning) and segmentation tokens (for grounding) of the same region.

Approach

Treats the visual prompt token <VP> (input mask in captioning) and the [SEG] token (output mask in grounding) as the same region's representation from opposite directions, and enforces consistency via a self-reconstruction loss (Stage 1) and a cosine latent loss (Stage 2), together with a hybrid region extractor combining SAM and CLIP-style features.

Key results

• FCLM-7B reaches 82.6 cIoU on RefCOCO val and 82.1 gIoU on ReasonSeg-like DL-RES/ReasonSeg column reported as 82.1 in Table 1 for the 7B model, outperforming UniPixel-7B.
• FCLM-3B achieves 52.0 Pos / 83.6 Neg / 67.8 Avg on DLC-Bench, comparable to DAM (67.3 Avg) and above GPT-4o (61.5 Avg).
• FCLM-7B reaches 51.5 gIoU / 57.4 cIoU on MUSE test, above PixelLM-13B (44.8 / 54.1).

Surprising findings

• Combining the self-reconstruction loss with the latent loss in Stage 2 leads to unstable pseudo-labels and lower performance than latent loss alone.
• Training the LLM on multiple visual tasks maintains and slightly improves performance, suggesting implicit cross-task correlation.

Limitations

• Limited to image inputs; extension to video temporal sequences is left to future work
• Spatial prompts limited to masks; boxes and points not supported
• DL-RES referring expressions were generated with DAM and manually refined, so the benchmark relies on manual effort

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg ReasonSeg MUSE Pixel2Cap Osprey Visual Genome LVIS-PACO GLaMM Describe Anything

Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg MUSE DL-RES LVIS PACO DLC-Bench GCG

Metrics: cIoU gIoU semantic similarity semantic IoU Pos/Neg/Avg accuracy METEOR CIDEr AP50 mIoU Mask Recall

Baselines: LISA VISA READ Sa2VA UniPixel PixelSAIL GLaMM DAM Osprey PixelLM

Builds on: Qwen2.5-VL LISA Osprey DAM SAM

Closest papers by meaning

• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.92)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.90)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
• ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.90)
• FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.90)
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want (ICLR 2025 · similarity 0.90)