AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

UniCode : Learning a Unified Codebook for Multimodal Large Language Models

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use a unified codebook instead of separate visual and text tokenizers (ablation)UniCode · ImageNet generationFID: 9.82 -> 6.72Improved
use a unified codebook instead of visual encoder plus text tokenizer (ablation)UniCode · VQAv2accuracy: 52.3 -> 53.1Improved
use language-driven iterative codebook training instead of frozen LLM codebook (ablation)UniCode · ImageNet generationFID: 34.45 -> 6.72Improved
use dual alternative training instead of language-driven iterative training (ablation)UniCode · VQAv2accuracy: 53.1 -> 9.3Hurt
add in-context image decompression pre-training task (ablation)UniCode · ImageNet generationFID: 7.08 -> 6.72Improved
upgrade visual tokenizer from VQ-GAN to RQ-VAE to HQ-VAE (ablation)UniCode · POPEaccuracy: 65.1 -> 71.8Improved
add extra CC3M images, ground-truth images, and a pretrained larger ViT to the visual encoder (ablation)UniCode · POPEaccuracy: 71.8 -> 77.6Improved
test at code map resolution larger than 256 used in pretraining (ablation)UniCode · VQAv2accuracy: 53.1 -> 41.3 (320)Hurt
pretrain at 320x320 to match test resolution (ablation)UniCode · VQAv2accuracy: 41.3 -> 54.5Improved
use UniCode instead of HQ-TVAE* of same 7B scaleimage generation · CC3M valFID: 12.13 -> 11.54Improved
use unified codebook versus HQ-VAE for reconstructionHQ-UniCode · ImageNet reconstructionrFID: 2.61 -> 2.83No clear effect
compare UniCode against LLaVA-1.5 on VQAUniCode · POPEaccuracy: 86.4 -> 71.8Hurt

Later papers that compared with this method

1 later accepted paper reports Unicode as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy