Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V2L (Vision-to-Language) Tokenizer that maps images into discrete tokens from a frozen LLM's vocabulary (expanded with bigrams/trigrams for global tokens), letting a frozen LLaMA-2 perform image classification, captioning, VQA and denoising via in-context learning without fine-tuning the LLM.
Problem
Enabling frozen LLMs to comprehend and restore visual signals without multi-modal fine-tuning or feature-space alignment, which prior methods require.
Approach
Treat images as a foreign language: an encoder-quantizer-decoder tokenizer quantizes local patch features and CLIP-based global features onto the frozen LLM vocabulary (global codebook expanded with bigrams/trigrams filtered by CLIP similarity), so a frozen LLM can process them via in-context prompts.
Key results
- On 5-way 5-shot-style Mini-ImageNet, LLaMA-2 70B with 5 tokens reaches 83.9 average accuracy vs. SPAE with PaLM-2 340B at 77.7 (2-way and 5-way settings reported).
- Reconstruction on ImageNet-1K: FID 3.41 with 256 tokens vs. SPAE 3.89 at 1109 tokens.
- Denoising across five tasks with LLaMA-2 70B yields lower FID than SPAE and LQAE on all tasks, e.g., deblurring FID 10.42 vs 18.08 for SPAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V2L Tokenizer instead of SPAE | Frozen LLaMA-2 70B · Mini-ImageNet 2-way, average | accuracy: 79.0 -> 83.9 | Helped |
| Use V2L Tokenizer instead of SPAE | Frozen LLaMA-2 70B · Mini-ImageNet 5-way, average | accuracy: 61.9 -> 81.5 | Helped |
| Use the expanded LLaMA-2 vocabulary instead of the PaLM-2 codebook | Image tokenization · ImageNet-1K val | CLIP: 0.1868 -> 0.2576 | Helped |
| Add global tokens to the local-token reconstruction setup (ablation) | V2L Tokenizer · ImageNet-1K val reconstruction | FID: 3.41 -> 2.88 | Helped |
| Add global tokens to the local-token reconstruction setup (ablation) | V2L Tokenizer · ImageNet-1K val reconstruction | PSNR: 23.56 -> 23.25 | Hurt |
| Use V2L Tokenizer instead of SPAE | LLaMA-2 70B denoising · Inpainting | FID: 12.79 -> 10.11 | Helped |
| Use V2L Tokenizer instead of SPAE | LLaMA-2 70B denoising · Outpainting | FID: 13.41 -> 10.73 | Helped |
| Use V2L Tokenizer instead of the best listed baseline | LLaMA-2 70B denoising · Deblurring | FID: 14.30 -> 10.42 | Helped |
| Use V2L Tokenizer instead of VQ-GAN | LLaMA-2 70B denoising · Rotation restoration | FID: 14.39 -> 10.48 | Helped |
| Use V2L Tokenizer instead of VQ-GAN | LLaMA-2 70B denoising · Shift restoration | FID: 14.35 -> 10.79 | Helped |
Limitations
- Global token pool relies on a filtered bigram/trigram vocabulary; semantic quality of expanded tokens can be unclear.
- Experiments are at low resolution (128x128) and on a limited set of tasks.
- Denoising and restoration evaluated with re-implemented baselines, so comparisons depend on re-implementation fidelity.
Experimental setup
Datasets: ImageNet-1K COCO Caption VQA Mini-ImageNet
Benchmarks: Mini-ImageNet COCO Caption VQA ImageNet-1K validation
Metrics: accuracy FID LPIPS PSNR CLIP score CLIP-R
Baselines: SPAE LQAE VQ-GAN Frozen
Builds on: VQ-GAN SPAE LQAE CLIP
Closest papers by meaning
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.92)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.91)
- ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs (AAAI 2026 · similarity 0.90)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.89)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.