Research map / Vision-language and multimodal / Multimodal LLMs
All in Tokens: Unifying Output Space of Visual Tasks via Soft Token
Jia Ning, Chen Li, Zheng Zhang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, Han Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use soft token in both task-solver and detokenizer instead of hard inference (ablation) | AiT · NYUv2 depth | RMSE: 0.3174 -> 0.3080 | Improved |
| use soft token in both task-solver and detokenizer (ablation) | AiT · COCO instance segmentation | mask mAP: 31.1 -> 33.2 | Improved |
| add auxiliary loss on detokenizer output (ablation) | AiT with soft token · COCO instance segmentation | mask mAP: 33.2 -> 34.2 | Improved |
| add auxiliary loss on detokenizer output (ablation) | AiT with soft token · COCO instance segmentation | box mAP: 43.6 -> 43.3 | Hurt |
| add soft token to parallel decoder (ablation) | AiT-P (SwinV2-L) · NYUv2 depth | RMSE: 0.282 -> 0.275 | Improved |
| add mask augmentation with ratio 0.5 in VQ-VAE training (ablation) | AiT · NYUv2 depth | RMSE: 0.3105 -> 0.3080 | Improved |
| increase mask ratio to 0.7 (ablation) | AiT · NYUv2 depth | RMSE: 0.3080 -> 0.3225 | Hurt |
| increase VQ-VAE codebook size from 64 to 256 (ablation) | AiT task-solver · NYUv2 depth | RMSE: 0.3090 -> 0.3119 | No clear effect |
| increase VQ-VAE width from 1.0x to 2.0x (ablation) | AiT task-solver · NYUv2 depth | RMSE: 0.3080 -> 0.3124 | No clear effect |
| reduce VQ-VAE downsample ratio from 32 to 8 for depth (ablation) | AiT · NYUv2 depth | RMSE: 0.3080 -> 0.3514 | Hurt |
| increase number of residual blocks in VQ-VAE from 2 to 3 or more (ablation) | AiT · NYUv2 depth | RMSE: 0.3080 -> 0.3123 | Hurt |
| replace VQ-VAE tokenizer with interpolation tokenizer (ablation) | AiT · COCO instance segmentation | mask mAP: 33.2 -> 4.2 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 2 of 2 on most shared results.
- Unified-IO XL — UNIFIED-IO: A Unified Model for Vision, Language, and Multi-modal Tasks (ICLR 2023) · better on 1 of 1 shared result
- UViM — UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (NeurIPS 2022) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.88)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.88)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.87)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.87)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.87)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
