Research map / Vision-language and multimodal / Multimodal LLMs

Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PaDT, which lets MLLMs output visual reference tokens (VRTs) taken from input patch embeddings and decodes them with a lightweight decoder into boxes, masks and scores for detection, segmentation, grounding and captioning.

Problem

Existing MLLMs express visual outputs as textual coordinates, which yields inconsistent formats, weak semantic alignment, fragmented number tokens, and cannot support dense prediction such as segmentation.

Approach

Reuse projected visual patch embeddings of the query image as dynamically added output vocabulary (VRTs) interleaved with text, with a PaDT head extending the classifier, a lightweight two-way-attention decoder producing boxes/masks/scores, and a robust per-token CE loss over randomly sampled foreground VRTs.

Key results

• PaDT Pro 3B reaches 93.6 average REC accuracy across RefCOCO/+/g, above the 78B InternVL3 (91.4).
• PaDT 3B obtains 34.0 COCO AP@[50:95] for open-vocabulary detection versus 13.7 for Qwen2.5-VL-3B.
• PaDT 3B achieves 76.1 RefCOCO val cIoU for referring segmentation, versus 74.1 for LISA-7B-level baselines reported in the table (PaDT 3B 76.1 vs Seg-R1 3B 69.9).

Surprising findings

• Using all foreground patches as VRT supervision collapses performance (RefCOCO val REC 49.1 with robust CE), suggesting dense supervision harms training.
• Adding SAM2 with point prompts does not improve over PaDT, while box or mask prompts do.

Limitations

• Performance drops sharply when all foreground patches are used as targets, requiring careful selection of VRT count (5 per target).
• Segmentation mask quality is bounded by low-resolution patch-level features; SAM2 refinement adds inference cost.
• RIC dataset is constructed with VLM-generated captions and not released per the text.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg COCO 2017 Objects365

Benchmarks: RefCOCO RefCOCO+ RefCOCOg COCO 2017 val RIC validation set

Metrics: accuracy@IoU0.5 cIoU AP@[50:95] AR@[50:95] CIDEr-D METEOR ROUGE-L BLEU-4 GreedyPrecision GreedyRecall

Baselines: Qwen2.5-VL InternVL3 VLM-R1 Seg-R1 Text4Seg ClawMachineX LISA OMG-LLaVA Ferret-v2 UNINEXT-H

Builds on: Qwen2.5-VL ClawMachine SAM2 Grounding DINO

Closest papers by meaning

• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.91)
• A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.91)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.90)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.90)