Research map / Vision-language and multimodal / Multimodal LLMs
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMG-LLaVA, a single-encoder, single-decoder MLLM that uses a frozen OMG-Seg universal perception module with perception prior embedding to handle image-level, object-level, and pixel-level understanding and segmentation in one model.
Problem
Existing MLLMs either lack pixel-level grounding and visual-prompt support, or achieve it via complex multi-encoder/multi-decoder systems (e.g., LLM agents, SAM-based pipelines) that are costly and lose general conversation ability.
Approach
Use a frozen universal segmentation model (OMG-Seg) as the sole visual encoder/decoder: its image features are fused with object queries via mask-score weighting (perception prior embedding) to form pixel-centric tokens, while foreground object queries serve as object-centric tokens; the LLM emits [SEG] tokens decoded by the same frozen decoder, and visual prompts (point/box/mask) are encoded as a
Key results
- Fine-tuned OMG-LLaVA reaches 78.0 / 69.1 / 72.9 cIoU on refCOCO / refCOCO+ / refCOCOg val, surpassing LISA by 3.1, 4.0, and 5.0 cIoU.
- Perception prior embedding raises refCOCO cIoU from 58.7 (baseline) to 72.5 and GCG mIoU from 51.0 to 62.1 in the ablation.
- On GranDf test, OMG-LLaVA (ft) reaches 38.5 CIDEr and 64.7 mIoU, outperforming LISA and approaching GLaMM which used more pretraining data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add perception prior embedding (fuse object queries into pixel features via mask-score weighted average) (ablation) | OMG-LLaVA with frozen OMG-Seg decoder · refCOCO / refCOCO+ / refCOCOg RES | cIoU: +13.8 refCOCO, +10.6 refCOCO+, +11.7 refCOCOg | Helped |
| add perception prior embedding (ablation) | OMG-LLaVA baseline (OMG-Seg + LLaVA) · GCG | mIoU: +11.1 mIoU | Helped |
| add object query input (foreground object queries as object-centric tokens) (ablation) | OMG-LLaVA with perception prior embedding · refCOCO RES and GCG | cIoU / mIoU: +1.9 cIoU refCOCO, +1.5 mIoU GCG | Helped |
| add cross-attention layer to object-centric token projector (ablation) | OMG-LLaVA projector · refCOCO RES | cIoU: -2.2 cIoU vs shared MLP baseline (74.5 to 72.3) | Hurt |
| use unshared (individual) projectors for object-centric tokens from visual prompts and object queries (ablation) | OMG-LLaVA projector · refCOCO RES | cIoU: -2.2 cIoU vs shared MLP baseline (74.5 to 72.3) | Hurt |
| unify segmentation answer format as '<p> Expression </p> [SEG]' instead of fixed 'It is [SEG].' (ablation) | OMG-LLaVA RES and GCG training · refCOCO RES | cIoU: +0.1 cIoU (75.5 to 75.6); also small gains on refCOCO+ and refCOCOg | Helped |
| use mean of all layers' [SEG] hidden states as segmentation embedding (ablation) | OMG-LLaVA · refCOCOg RES | cIoU: -1.3 cIoU (70.0 to 68.7) on refCOCOg val | Hurt |
| concatenate all layers' [SEG] hidden states as segmentation embedding (ablation) | OMG-LLaVA · RES (refCOCO, refCOCO+, refCOCOg) | cIoU: -4.3 cIoU refCOCO (74.3 to 70.0) | Hurt |
| joint training with pixel-level segmentation data (vs LLaVA data only) (ablation) | OMG-LLaVA image-level benchmarks · MMBench / SEED / MME | MMBench: 67.5 (LLaVA-only) vs 47.9 (co-trained) | Hurt |
| use Qwen2-7B or Phi3-3.8B instead of InternLM2-7B as LLM (ablation) | OMG-LLaVA · refCOCO / refCOCO+ RES and image benchmarks | cIoU: Qwen2-7B 76.7 / 69.1 vs InternLM2-7B 76.3 / 67.7 on refCOCO / refCOCO+ | Helped |
| finetune OMG decoder (unfreeze) on RES | OMG-LLaVA · refCOCO / refCOCO+ / refCOCOg val | cIoU: +2.4 refCOCO (75.6 to 78.0), +3.5 refCOCO+ (65.6 to 69.1), +2.2 refCOCOg (70.7 to 72.9) | Helped |
| finetune on GranDf for grounded conversation generation | OMG-LLaVA · GranDf test | CIDEr: 33.1 to 38.5 CIDEr (+5.4) | Helped |
Surprising findings
- Joint training with pixel-level data reduces image-level capability; OMG-LLaVA's MMBench (47.9) is far below LLaVA 1.5 (68.5) in the co-training setting.
- Concatenating or averaging hidden states across all LLM layers for the [SEG] embedding does not help; the last-layer state is sufficient.
Limitations
- Joint training with pixel-level data degrades image-level conversation and benchmark performance.
- OMG-Seg lacks multi-granularity segmentation, so part-level segmentation is not supported.
- No pixel-level spatial-temporal (video) reasoning capability.
Experimental setup
Datasets: LLaVA pretraining data LLaVA instruction data Osprey MDVP refCOCO refCOCO+ refCOCOg refClef ADE20k COCO-stuff GranDf
Benchmarks: COCO panoptic segmentation VIPSeg refCOCO refCOCO+ refCOCOg GranDf GCG MME MMBench SEED-Bench POPE AI2D
Metrics: PQ VPQ cIoU gIoU METEOR CIDEr AP50 mIoU
Baselines: LISA GLaMM PixelLM GSVA LaSagnA AnyRef Kosmos-2 Osprey LLaVA 1.5 NeXT-Chat
Builds on: LLaVA LISA OMG-Seg GLaMM xtuner
Closest papers by meaning
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.90)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.88)
- TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings (AAAI 2025 · similarity 0.88)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.88)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.