Research map / Large language models / Language, safety and interpretability
Making LLaMA SEE and Draw with SEED Tokenizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SEED, a discrete image tokenizer with 1D causal dependency and high-level semantics, and SEED-LLaMA, an LLM trained with next-token prediction over interleaved image-text tokens for multimodal comprehension and generation.
Problem
Existing visual tokenizers (e.g., VQ-VAE) capture low-level detail and use 2D dependencies incompatible with left-to-right LLM autoregression, so MLLMs struggle to unify comprehension and generation with the original next-word-prediction recipe.
Approach
A VQ tokenizer whose Causal Q-Former produces 32 causally dependent query embeddings trained with contrastive learning for high-level semantics, then quantized and decoded to a unCLIP-SD generation embedding; the discrete codes are added as new vocabulary words to a pretrained LLM trained with unified next-token prediction.
Key results
- SEED-LLaMA-I 14B reaches 126.9 CIDEr on COCO captioning, versus 61.6 for CM3Leon, and 45.2 on SEED-Bench, the best among compared models.
- SEED causal embeddings match or exceed BLIP-2 on COCO and Flickr30K retrieval (Recall@mean 82.5 vs 80.3 on COCO).
- SEED-LLaMA-I 8B achieves 70.68 CLIP-similarity on COCO text-to-image generation, above GILL (67.45) and Emu (66.46).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply multimodal instruction tuning | SEED-LLaMA, 8B · VizWiz | VQA accuracy: 21.5 -> 55.1 | Helped |
| Apply multimodal instruction tuning | SEED-LLaMA, 8B · COCO text-to-image | CLIP similarity: 69.07 -> 70.68 | Helped |
| Use the larger LLaMA2-Chat-13B backbone instead of Vicuna-7B | SEED-LLaMA-I · SEED-Bench | MCQ accuracy: 51.5 -> 53.7 | Helped |
| Align generation embeddings with unCLIP-SD image embeddings rather than SD text embeddings (ablation) | SEED tokenizer · COCO image-to-image reconstruction | CLIP similarity: 68.23 -> 77.35 | Helped |
| Use causal visual codes instead of bilateral visual codes (ablation) | SEED-LLaMA* · COCO text-to-image generation | successful image generations: 2134 -> 4997 | Helped |
| Add full fine-tuning after LoRA tuning (ablation) | SEED-LLaMA · COCO image captioning | CIDEr: 124.5 -> 125.0 | Helped |
| Add full fine-tuning after LoRA tuning (ablation) | SEED-LLaMA · COCO text-to-image | CLIP similarity: 68.87 -> 69.07 | Helped |
| Quantize causal embeddings into discrete causal codes (ablation) | SEED tokenizer · Flickr30K image-text retrieval | R@m: 93.6 -> 90.8 | Hurt |
| Quantize causal embeddings into discrete causal codes (ablation) | SEED tokenizer · COCO image-text retrieval | R@m: 82.5 -> 78.2 | Hurt |
Surprising findings
- Non-causal bilateral visual codes produced only 2134 of 5000 COCO images successfully versus 4997 for causal codes, indicating instability without causal dependency.
- Fully fine-tuning the LLM after LoRA pretraining gives only small gains (CIDEr 124.5 to 125.0), and freezing the embedding layer was needed for stable training.
Limitations
- Visual tokenizer reconstruction is below the unCLIP-SD upper bound (CLIP similarity 77.35 vs 79.30 on COCO).
- Video QA and VQA results lag continuous-feature MLLMs such as Emu-I and InstructBLIP on several benchmarks (e.g., VizWiz and MSRVTT-QA).
Experimental setup
Datasets: CC3M Unsplash LAION-COCO MS-COCO MMC4 OBELISC WebVid JourneyDB DiffusionDB LAION-Aesthetics VIST InstructPix2Pix
Benchmarks: COCO Caption VQAv2 OKVQA VizWiz SEED-Bench MSVDQA MSRVTTQA NExTQA COCO text-to-image Flickr30K text-to-image Flickr30K retrieval COCO retrieval
Metrics: CIDEr VQA accuracy MCQ accuracy top-1 accuracy WUPS CLIP score Recall@K
Baselines: Flamingo BLIP-2 InstructBLIP Kosmos-1 Kosmos-2 CM3Leon Emu Emu-I GILL IDEFICS unCLIP SD
Builds on: BLIP-2 unCLIP Stable Diffusion VQ-VAE LoRA
Closest papers by meaning
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
- Auto-Encoding Morph-Tokens for Multimodal LLM (ICML 2024 · similarity 0.87)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.87)
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.