Research map / Vision-language and multimodal / Multimodal LLMs

DenseMLLM: Standard Multimodal LLMs for Dense Prediction

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DenseMLLM, a standard ViT-projector-LLM MLLM that produces semantic segmentation, depth, and referring masks directly from vision-token logits, trained with a multi-label vision next-token objective (NTP-M) with relevant negative sampling.

Problem

Extending MLLMs to fine-grained dense prediction (segmentation, depth, referring segmentation) usually requires task-specific decoders or extra tokens, fragmenting the generalist architecture.

Approach

Supervise vision tokens with a multi-hot, multi-label next-token objective over a unified vocabulary (independent sigmoid per vocab entry), with positives averaged separately from top-k relevant negatives, and read dense predictions by argmax over vision-token logits for category token IDs; no extra decoder.

Key results

• Achieves 54.2 mIoU on ADE20K without task-specific decoders, vs 52.3 for VisionLLM-v2 and 47.8 for GiT.
• Reaches 80.7 cIoU on RefCOCO val, above UFO (80.0) and GLaMM (79.5), though below UniPixel (80.5) on that split.
• Ablation on ADE20K: NTP-M with relevant negative sampling raises mIoU from 32.7 (individual mean) to 51.2, while standard NTP reaches 35.33.

Surprising findings

• Standard single-label NTP with Focal Loss or OHEM degrades dense prediction, while the multi-label loss with relevant negatives works best.
• Vision-token supervision also improves PCA visualization quality of vision token hidden states compared with Qwen3-VL.

Limitations

• Performance on rare, long-tail open-world categories is constrained by public training data diversity.
• Focuses on semantic-level tasks; does not address instance or panoptic segmentation, surface normals, or optical flow.
• Large proprietary pretraining data (~11.8T tokens) is undisclosed, hindering reproducibility.

Experimental setup

Datasets: ADE20K COCO-Stuff Pascal Context Cityscapes Pascal VOC RefCOCO NYUv2 DDAD

Benchmarks: ADE20K COCO-Stuff Pascal Context Cityscapes Pascal VOC NYUv2 DDAD RefCOCO RefCOCO+ RefCOCOg MMBench MMStar

Metrics: mIoU delta1 cIoU accuracy

Baselines: Segformer Mask2Former UniDepth-v2 SwinMTL RVG VPD SAN GLaMM UniPixel VisionLLM-v2 UFO GiT

Builds on: DAPO Qwen2.5-VL SigLIP-2 VisionLLM-v2 DepthLM

Closest papers by meaning

• Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.91)
• DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
• Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.90)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.89)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)