Research map / Vision-language and multimodal / Multimodal LLMs
UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPixel, a Qwen2.5-VL-based LMM (3B/7B) that unifies visual-prompt referring and mask segmentation via an object memory bank and SAM 2.1 mask decoder, and introduces the PixelQA task.
Problem
Existing LMMs handle referring or segmentation separately with rigid templates and cannot integrate pixel-level perception into multi-modal visual reasoning in images and videos.
Approach
An object memory bank (hashmap of object IDs to spatial-temporal masks) pre-filled from <REF>/<SEG> tokens and injected back as masked-pooled features, combined with a prompt encoder that embeds points/boxes with temporal Fourier encodings and a SAM 2.1 decoder, so one model refers, segments and reasons end to end.
Key results
- On ReVOS, the 3B UniPixel reaches 62.1 J&F, and the 7B reaches 63.7 overall J&F in the main table (text cites 64.0 for 7B).
- On VideoRefer-Bench-Q with multi-frame mask prompts, the 3B model achieves 72.8% accuracy, above VideoRefer-7B at 72.1.
- On MeViS, the 3B model outperforms GLUS-7B by about 3.5 points in J&F on the validation split (53.1 vs 51.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| unify referring and segmentation in joint training (ablation) | UniPixel 3B · PixelQA mixed prompts | Acc: 64.6 -> 67.4 | Helped |
| add memory pre-filling to unified referring and segmentation (ablation) | UniPixel 3B · PixelQA mixed prompts | Acc: 67.4 -> 68.5 | Helped |
| use object memory bank instead of <REF> with auxiliary <SEG> and pooled tokens (ablation) | UniPixel 3B · PixelQA mixed prompts | Acc: 66.3 -> 68.5 | Helped |
| add temporal encoding to the prompt encoder (ablation) | UniPixel 3B · PixelQA mixed prompts | J&F: 44.3 -> 49.0 | Helped |
| propagate masks across video frames rather than decode frames independently (ablation) | UniPixel 3B · PixelQA mixed prompts | J&F: 46.1 -> 49.0 | Helped |
| use 4 hidden tokens rather than 2 per SEG token (ablation) | UniPixel 3B · MeViS valᵘ | J&F: 59.7 -> 59.9 | No clear effect |
| pre-train the sparse prompt encoder (ablation) | UniPixel 3B · ReVOS | J&F: 61.0 -> 61.2 | Helped |
| pre-train the L-to-M projector (ablation) | UniPixel 3B · ReVOS | J&F: 61.0 -> 61.6 | Helped |
| pre-train both the sparse prompt encoder and L-to-M projector (ablation) | UniPixel 3B · ReVOS | J&F: 61.0 -> 62.1 | Helped |
| initialize M-to-L projector from the pre-trained V-to-L projector (ablation) | UniPixel 3B · VideoRefer-Bench Q single-frame | accuracy: 71.4 -> 72.4 | Helped |
| add extra M-to-L projector pre-training after weight initialization (ablation) | UniPixel 3B · PixelQA mixed prompts | Acc: 68.2 -> 68.5 | Helped |
| add memory pre-filling data to regional and segmentation stage-3 training (ablation) | UniPixel 3B · ReVOS | J&F: 61.5 -> 62.1 | Helped |
Surprising findings
- Reusing the pre-trained V-to-L projector weights for the M-to-L projector without extra pre-training gives the best results in the ablation.
- Using more than 2 hidden tokens per <SEG> gives negligible gains, and treating video frames independently for mask decoding is sub-optimal.
Experimental setup
Datasets: Inst-IT VideoRefer RefCOCO RefCOCO+ RefCOCOg RefClef Ref-YouTube-VOS MeViS ReVOS ReasonSeg Ref-SAV GroundMoRe
Benchmarks: ReVOS MeViS Ref-YouTube-VOS Ref-DAVIS17 Ref-SAV GroundMoRe RefCOCO RefCOCO+ RefCOCOg ReasonSeg VideoRefer-Bench-D VideoRefer-Bench-Q
Metrics: J&F cIoU gIoU accuracy GPT-4o score
Baselines: LISA VISA Sa2VA GLUS VideoLISA VideoRefer GPT-4o VideoGLaMM
Builds on: Qwen2.5-VL SAM 2.1 LISA Sa2VA
Closest papers by meaning
- OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding (NeurIPS 2024 · similarity 0.90)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.90)
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.89)
- Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.