Research map / Large language models / Language, safety and interpretability
Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
Jindong Li, Yali Fu, Jiahong Liu, L. J. Cao, Wei Ji, Menglin Yang, Irwin King, Ming–Hsuan Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FSQ instead of VQGAN quantizer | Image tokenizer · ImageNet generation 128x128 | gFID: 8.01 -> 7.33 | Improved |
| use BSQ instead of VQGAN quantizer | Image tokenizer · ImageNet reconstruction 256x256 | rFID: 1.228 -> 0.700 | Improved |
| use FSQ instead of VQGAN quantizer | Image tokenizer · ImageNet generation 128x128 | IS: 151.40 -> 224.88 | Improved |
| use LFQ instead of VQGAN quantizer | Image tokenizer · ImageNet reconstruction 256x256 | rFID: 1.228 -> 2.444 | Hurt |
| use LFQ instead of VQGAN quantizer | Image tokenizer · ImageNet generation 128x128 | gFID: 8.01 -> 7.73 | Improved |
| use RVQ/CQ/AQ multi-codebook full-dimensional quantization instead of PQ subspace splitting | ANN search · SIFT1M ANN search (64-bit codes) | Recall@1: 0.224 -> 0.298 | Improved |
| use AQ instead of PQ | ANN search · SIFT1M ANN search (64-bit codes) | Recall@10: 0.599 -> 0.741 | Improved |
| use AQ instead of PQ | ANN search · SIFT1M ANN search (64-bit codes) | Recall@100: 0.924 -> 0.972 | Improved |
| use RVQ instead of PQ | ANN search · SIFT1M ANN search (64-bit codes) | Recall@1: 0.224 -> 0.259 | Improved |
Closest papers by meaning
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.91)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.90)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.89)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.89)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.89)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
