Research map / Vision-language and multimodal / Multimodal LLMs

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a BPE image tokenizer that merges VQ-GAN token IDs into larger 2D visual tokens, supported by a theoretical argument and a preliminary Llama-3.1-8B MLLM (Being-VL-0) showing gains over a VQ-only baseline.

Problem

Token-based multimodal LLMs feed raw quantized image tokens directly to Transformers, which struggle to learn 2D image structure and align with text, and they consume large compute for limited gains.

Approach

Apply byte-pair-encoding-style merging to 2D quantized image tokens (using horizontal and vertical co-occurrence counts) so that tokens carry structural priors, and show theoretically that a tokenizer can approach the optimal unconstrained entropy for 2D Markov data whereas unigram models cannot.

Key results

• Being-VL-0 (LLM+VQ+BPE, PT(freeze)+SFT) reaches 57.1 VQAv2, 40.9 MMBench, 1223.5 MME-p and 79.0 POPE versus 55.4, 37.6, 1054.5 and 76.0 for the LLM+VQ baseline.
• Adding more PT and SFT data (RefCOCO, AOKVQA, ShareGPT4o, ALLaVA) raises VQAv2 to 60.6 and MMBench to 44.0.
• On MLLM-bench the BPE variant wins 173 answers versus 149 for the VQ-only baseline, with 98 ties.

Measured effects reported in the paper

Surprising findings

• Freezing text token embeddings during pretraining outperforms updating all embeddings (e.g., MME-p 1223.5 vs 1144.6).
• Vocabulary size of 16K degrades most benchmarks relative to 8K, and the 8K BPE setting uses tokens more uniformly.

Limitations

• Experiments use limited compute and training data and do not match state-of-the-art MLLMs.
• The theoretical analysis relies on a simplified 2D Markov process that may not capture real image statistics.
• Captioning results show LLM+VQ+BPE slightly below LLM+VQ before SFT.

Experimental setup

Datasets: ImageNet CC-3M LAION SBU LCS LLaVA-OneVision RefCOCO AOKVQA ShareGPT4o ALLaVA Inst

Benchmarks: VQAv2 MMBench MME POPE VizWiz MLLM-bench Nocaps Flickr30k

Metrics: accuracy MME perception score MME cognition score CIDEr SPICE METEOR ROUGE-L win/tie counts

Baselines: LLM+VQ Llama2+VQ Llama2+VQ+BPE

Builds on: byte-pair encoding VQ-GAN Chameleon LLaVA

Closest papers by meaning

• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.91)
• A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.91)
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.91)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.90)