Research map / Vision-language and multimodal / Multimodal LLMs
PixelLM: Pixel Reasoning with Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Rebuttal-style response for PixelLM, a large multimodal model that generates pixel-level masks via a lightweight pixel decoder and a segmentation codebook of special tokens, with added ablations on MUSE, LLaVA-Bench and discrete-token alternatives.
Problem
Generating pixel-level masks for image reasoning involving multiple open-world targets remains challenging for large multimodal models.
Approach
A lightweight pixel decoder plus a codebook of shared [SEG] tokens (N=6) that the LLM autoregressively generates; the hidden embeddings condition the decoder to produce masks for multiple targets, avoiding reliance on SAM.
Key results
- Training with MUSE raises LISA's ReasonSeg val gIoU from 52.9 to 58.0 and cIoU from 54.0 to 59.1.
- PixelLM with CLIP-H reaches 72.6 cIoU on refCOCOg and 47.8 cIoU on MUSE, improving over the default setting.
- Discrete coordinate tokens with 128 polygon points per mask perform worse than PixelLM across datasets (e.g., 59.1 vs 69.3 on refCOCOg).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MUSE training data to LISA (ablation) | LISA · ReasonSeg val | gIoU: 52.9 -> 58.0 | Helped |
| add MUSE training data to LISA (ablation) | LISA · ReasonSeg val | cIoU: 54.0 -> 59.1 | Helped |
| replace CLIP vision encoder with laion-CLIP-H (ablation) | PixelLM · refCOCOg / MrefCOCOg / MUSE | cIoU: 69.3->72.6 (refCOCOg), 65.8->68.0 (MrefCOCOg), 46.3->47.8 (MUSE) | Helped |
| predict 128 polygon points with discrete coordinate tokens instead of compact segmentation tokens (ablation) | PixelLM · refCOCOg / MrefCOCOg / MUSE | cIoU: 69.3->59.1 (refCOCOg), 65.8->46.8 (MrefCOCOg), 46.3->36.2 (MUSE) | Hurt |
| change loss weights (lambda_ref, lambda_dice) from default 2/0.5 to 1/1 or 0.5/2 (ablation) | PixelLM · refCOCOg / MrefCOCOg / MUSE | cIoU: 69.3->68.8/68.6 (refCOCOg) | Hurt |
| add MUSE text-only image-text data (no mask loss) to LLaVA training (ablation) | LLaVA · LLaVA-Bench All | score: 62.7 -> 63.1 | Helped |
| add MUSE data with segmentation mask/loss to LLaVA training (ablation) | LLaVA · LLaVA-Bench All | score: 62.7 -> 63.3 | Helped |
| train with segmentation decoder and mask targets (vs. text-only MUSE) (ablation) | LLaVA · LLaVA-Bench All | score: 63.1 -> 63.3 | Helped |
| use the 15k-sample MUSE‡ test set instead of MUSE val | PixelLM · MUSE | cIoU: 46.3 -> 45.9 | No clear effect |
| add a comprehensive segmentation codebook plus lightweight pixel decoder (vs. LISA SAM-based decoder) | PixelLM · refCOCO family | cIoU | Helped |
Surprising findings
- PixelLM scores lower than LLaVA on LLaVA-Bench (51.7 vs 62.7 All), attributed to the brief, simple text in segmentation datasets.
- Decoder alone without the codebook surpasses the SAM-based decoder in cIoU but trails slightly in gIoU.
Limitations
- Segmentation datasets used for training carry low-quality text, hurting general VL performance.
- Discrete-token mask generation was worse and long polygon sequences cause autoregressive errors; combining VQGAN-like tokens with LLMs is left open.
Experimental setup
Datasets: MUSE LLaVA-150k ReasonSeg refCOCO refCOCOg MrefCOCOg ade20k
Benchmarks: ReasonSeg refCOCOg MrefCOCOg MUSE LLaVA-Bench
Metrics: cIoU gIoU
Baselines: LISA UNINEXT Pix2Seq v2 VisionLLM LLaVA
Builds on: LISA SAM LLaVA Pix2Seq v2 VisionLLM
Closest papers by meaning
- SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories (CVPR 2025 · similarity 0.90)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.89)
- PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images (ICLR 2026 · similarity 0.89)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.88)
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.88)
- MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.