Research map / Vision-language and multimodal / Multimodal LLMs
ObjecTok: Learning Holistic and Robust Object Tokens for MLLMs
Sihan Wang, Xiyao Liu, Lianqing Liu, Zhi Han
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ObjecTok, a framework that represents each object in an image with a single holistic object token instead of patch tokens, aiming to improve MLLM perception and reduce object hallucination.
Problem
Patch-based visual tokenization in multimodal large language models breaks object integrity, limiting perception and causing object-related hallucinations.
Approach
Generate one holistic token per object in an image so that each object is represented as an integral unit rather than fragmented across patches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add object tokens to LLaVA-1.5 | ObjecTok-LLaVA · POPE (total) | F1: 85.95 -> 88.31 | Improved |
| add geometric decoder in object encoder training (ablation) | ObjecTok-LLaVA · MME Position | score: 113.33 -> 140.00 | Improved |
| add geometric decoder in object encoder training (ablation) | ObjecTok-LLaVA · MME Count | score: 165.00 -> 178.33 | Improved |
| add confidence embeddings (ablation) | ObjecTok-LLaVA · MME Existence | score: 145.00 -> 195.00 | Improved |
| add confidence embeddings (ablation) | ObjecTok-LLaVA · MME Position | score: 140.00 -> 140.00 | No clear effect |
| add patch tokens alongside object tokens (ablation) | ObjecTok-LLaVA · MME Commonsense Reasoning | score: 93.57 -> 137.86 | Improved |
| add patch tokens alongside object tokens (ablation) | ObjecTok-LLaVA · MME Color | score: 88.33 -> 170.00 | Improved |
| add object tokens to LLaVA-1.5 | ObjecTok-LLaVA · POPE Adversarial | F1: 84.46 -> 85.60 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- MiniGPT-4 — MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Grounding Everything in Tokens for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.88)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
