Research map / Vision-language and multimodal / Multimodal LLMs
Unified Multimodal Understanding via Byte-Pair Visual Encoding
Wanpeng Zhang, Yicheng Feng, Hao Luo, Yijiang Li, Zihao Yue, Sipeng Zheng, Zongqing Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BPE visual vocabulary on top of VQ tokens (ablation) | Being-VL-0.5 · MMBench | accuracy: 38.2 -> 71.8 | Improved |
| add BPE visual vocabulary on top of VQ tokens (ablation) | Being-VL-0.5 · VQAv2 | accuracy: 54.3 -> 80.2 | Improved |
| add BPE visual vocabulary on top of VQ tokens (ablation) | Being-VL-0.5 · MME-P | MME-P score: 1301.2 -> 1525.8 | Improved |
| use priority-guided encoding and optimized training instead of frequency-only BPE (Being-VL-0) | Being-VL-0.5 · MMBench | accuracy: 44.0 -> 71.8 | Improved |
| use priority-guided encoding and optimized training instead of frequency-only BPE (Being-VL-0) | Being-VL-0.5 · MME-P | MME-P score: 1316.2 -> 1525.8 | Improved |
| increase BPE vocabulary from 8K to 16K (ablation) | Being-VL-0.5 · POPE | accuracy: 84.3 -> 86.0 | Mixed |
| add curriculum-based data composition and progressive unfreezing (vs single stage) (ablation) | Being-VL-0.5 · Perception avg score | avg score: 71.2 -> 80.3 | Improved |
| add curriculum-based data composition (vs progressive only) (ablation) | Being-VL-0.5 · Perception avg score | avg score: 74.9 -> 80.3 | Improved |
| add progressive parameter unfreezing (vs curriculum only) (ablation) | Being-VL-0.5 · Perception avg score | avg score: 76.8 -> 80.3 | Improved |
| use foundation-to-instruction curriculum instead of reverse ordering (ablation) | Being-VL-0.5 · Perception avg score | avg score: 73.4 -> 80.3 | Improved |
| use foundation-to-instruction curriculum instead of random mixing (ablation) | Being-VL-0.5 · Reasoning avg score | avg score: 68.4 -> 71.1 | Improved |
| add selective fine-tuning stage (3-stage vs 2-stage unfreezing) (ablation) | Being-VL-0.5 · Perception avg score | avg score: 78.2 -> 80.3 | Improved |
Closest papers by meaning
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.86)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.86)
- Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (ICLR 2025 · similarity 0.84)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.84)
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.84)
- MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
