Research map / Vision-language and multimodal / Multimodal LLMs
HyperSeg: Hybrid Segmentation Assistant with Fine-grained Visual Perceiver
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperSeg, a VLLM-based universal segmentation model for image and video perception that adds hybrid entity recognition, a fine-grained visual perceiver, and a temporal adapter, trained once across many segmentation and reasoning tasks.
Problem
Existing unified segmentation methods struggle to handle complex text instructions, reasoning segmentation, and both image and video domains within one model, and VLLM-based segmenters lack fine-grained visual detail and temporal modeling.
Approach
Combines a VLLM that first generates object names and then mask tokens (hybrid entity recognition, with class scores decoded from prompt embeddings), a multi-scale pyramid-feature perceiver that fuses high-resolution features into fine-grained tokens, and a temporal adapter with global prompt pooling and local space-time token propagation.
Key results
- HyperSeg reaches 79.0 cIoU on RefCOCO+ val (about +6.8 over PSALM's 72.9 per the text) and 84.8 on RefCOCO val.
- On ReVOS-Reasoning, HyperSeg-3B reaches 53.0 J&F, surpassing VISA-13B's 40.9 by about 12.1 points.
- On COCO-Panoptic, HyperSeg reaches 61.2 PQ versus 55.9 for PSALM, and 64.6 mIoU on PAS20-OV-style open-vocabulary evaluation (text reports +10.6 mIoU over PSALM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyperSeg instead of PSALM | Referring expression segmentation · RefCOCO+ val | score: 72.9 -> 79.0 | Helped |
| Use HyperSeg instead of VISA-13B | Video reasoning segmentation · ReVOS-Reasoning | J &F: 40.9 -> 53.0 | Helped |
| Use HyperSeg instead of OneFormer | Panoptic segmentation · COCO-Panoptic | PQ: 57.9 -> 61.2 | Helped |
| Use HyperSeg instead of UNINEXT | Video instance segmentation · YouTube-VIS 2019 | mAP: 64.3 -> 53.8 | Hurt |
| Add FVP and hybrid entity recognition (ablation) | HyperSeg · YT-VIS | mAP: 48.4 -> 53.8 | Helped |
| Add FVP and hybrid entity recognition (ablation) | HyperSeg · COCO-Panoptic | PQ: 54.8 -> 61.2 | Helped |
| Use multi-layer FVP instead of single-layer FVP (ablation) | HyperSeg · YT-VIS | mAP: 49.7 -> 50.4 | Helped |
| Add conditional weight to multi-layer FVP (ablation) | HyperSeg · YT-VIS | mAP: 50.4 -> 53.8 | Helped |
| Add global prompt aggregation (ablation) | HyperSeg temporal adapter · Ref-DAVIS17 | J &F: 67.3 -> 68.8 | Helped |
| Add local space-time information injection (ablation) | HyperSeg temporal adapter · Ref-DAVIS17 | J &F: 67.3 -> 69.3 | Helped |
| Add both global prompt aggregation and local space-time information injection (ablation) | HyperSeg temporal adapter · Ref-DAVIS17 | J &F: 67.3 -> 71.2 | Helped |
| Add image training datasets to video-task training (ablation) | HyperSeg · ReVOS-Overall | J &F: 54.1 -> 55.7 | Helped |
Surprising findings
- Joint training across tasks improves performance over task-specific models, and adding image data notably boosts video segmentation performance.
Limitations
- Video processed frame-by-frame, which may limit true temporal modeling.
- Gains on gRefCOCO and some open-vocabulary benchmarks remain modest in absolute terms.
Experimental setup
Datasets: COCO Panoptic RefCOCO RefCOCO+ RefCOCOg COCO-Interactive ReasonSeg DAVIS-2017 Ref-YouTube-VOS YouTube-VIS 2019 ReVOS LLaVA-150k
Benchmarks: RefCOCO RefCOCO+ RefCOCOg gRefCOCO ReasonSeg ReVOS COCO-Panoptic ADE-OV Citys-OV PC59-OV PAS20-OV DAVIS17
Metrics: cIoU gIoU PQ mIoU J&F mAP
Baselines: LISA PSALM OMG-LLaVA GLaMM VISA PixelLM UNINEXT OMG-Seg LaSagnA
Builds on: LISA PSALM Mask2Former Mipha
Closest papers by meaning
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.87)
- OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding (NeurIPS 2024 · similarity 0.87)
- Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design (CVPR 2026 · similarity 0.86)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.86)
- OMG-Seg: Is One Model Good Enough For All Segmentation? (CVPR 2024 · similarity 0.86)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.