Research map / Vision-language and multimodal / Multimodal LLMs
PerceptionGPT: Effectively Fusing Visual Perception into LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PerceptionGPT, which encodes boxes and masks as a single LLM token embedding decoded by lightweight encoders/decoders, LoRA-tuned on Vicuna, reaching SOTA on REC and RES with fewer trainable parameters.
Problem
Vision LLMs struggle with perception tasks like detection and segmentation because existing end-to-end approaches encode boxes and masks as long sequences of discrete tokens, which is hard to train, quantization-lossy, and slow to decode.
Approach
Use a single special <vis> token whose LLM output embedding carries a continuous box or mask, decoded by lightweight task decoders (MLP for boxes, two-way transformer for masks) and trained with L1+GIoU and BCE+DICE losses alongside the language modeling loss, plus learned multi-layer ViT feature fusion.
Key results
- On RefCOCO val, PerceptionGPT-13B reaches 89.17 vs Shikra-13B 87.83 on REC using LoRA training.
- On RES, PerceptionGPT-7B reaches 75.1 on RefCOCO val versus LISA 74.9 and the best prior end-to-end method ReLA 73.8.
- Decoding a mask with a single token embedding takes 0.15 s-scale time versus 42.1 for numerical representation at 7B (per Table of inference times), about 0.3% of numerical decoding time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use continuous token embeddings instead of numerical box tokens with LoRA rank 8 (ablation) | PerceptionGPT · RefCOCO val (limited training data) | Acc@0.5: 0.79 -> 69.7 | Helped |
| increase continuous-embedding LoRA rank from 8 to 32 (ablation) | PerceptionGPT · RefCOCO val (limited training data) | Acc@0.5: 69.7 -> 71.2 | Helped |
| increase continuous-embedding LoRA rank from 32 to full tuning (ablation) | PerceptionGPT · RefCOCO val (limited training data) | Acc@0.5: 71.2 -> 75.5 | Helped |
| increase numerical-token LoRA rank from 8 to 32 (ablation) | numerical box-token variant · RefCOCO val (limited training data) | Acc@0.5: 0.79 -> 10.5 | Helped |
| increase numerical-token LoRA rank from 32 to full tuning (ablation) | numerical box-token variant · RefCOCO val (limited training data) | Acc@0.5: 10.5 -> 45.7 | Helped |
| use one continuous token embedding rather than numerical representation (ablation) | PerceptionGPT · 7B box decoding | inference time: 3.62 -> 0.15 | Helped |
| use one continuous token embedding rather than numerical representation (ablation) | PerceptionGPT · 7B mask decoding | inference time: 42.1 -> 0.15 | Helped |
| use continuous token embeddings rather than discrete numerical tokens (ablation) | PerceptionGPT · RefCOCOg REC, limited training data, LoRA rank 8 | Acc@0.5 | Helped |
| adaptive multi-layer visual feature fusion (ablation) | PerceptionGPT · visual perception and image captioning tasks | performance | Helped |
Surprising findings
- Numerical box representations collapse under LoRA with low rank (e.g., 0.79 Acc at rank 8) while continuous token embeddings keep about 70%.
Limitations
- Only bounding box and mask tasks are demonstrated; other perception tasks such as depth or pose are not evaluated.
- Masks for Visual Genome and Flickr30k are auto-labelled by SAM, introducing label noise.
- Reasoning-based detection and segmentation emerged without targeted training data and are only shown qualitatively.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Visual Genome Flickr30k COCO LLaVA caption data
Benchmarks: RefCOCO RefCOCO+ RefCOCOg VQAv2 OK-VQA Flickr30k COCO
Metrics: Acc@0.5 mask IoU accuracy CIDEr inference time
Baselines: Shikra Kosmos-2 VisionLLM LISA UNINEXT-H Unified-IO OFA
Builds on: LLaVA SAM Shikra
Closest papers by meaning
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.90)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.90)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.89)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.88)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.88)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.