Research map / Vision-language and multimodal / Human and embodied modeling
Frozen Transformers in Language Models Are Effective Visual Encoder Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a single frozen pre-trained LLM transformer block (LLaMA-7B, OPT) inserted into visual encoders improves image, point cloud, video, motion forecasting and vision-language performance, and proposes an information filtering hypothesis to explain it.
Problem
Whether LLMs trained only on text can serve as useful encoders for purely visual tasks without language inputs or outputs, and why this would help.
Approach
Insert a single frozen transformer block from a pre-trained LLM (e.g., LLaMA-7B last block) between a visual encoder and its decoder, with two trainable linear layers for dimension alignment, removing causal masks and positional embeddings; attributes gains to an information filtering effect that amplifies informative visual tokens.
Key results
- ViT-S on ImageNet improves from 80.1 to 80.7 with a frozen LLaMA block, and ViT-B from 80.6 to 81.7.
- On SSv2 action recognition, ViT-B accuracy rises from 64.97 to 66.03 Acc1 with the LLaMA block.
- A randomly initialized LLM block (same parameter count) degrades ImageNet accuracy to 76.9, versus 80.7 with the frozen pre-trained block.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a frozen LLaMA block to ViT-T | ViT-T · ImageNet | accuracy: 72.1 -> 73.2 | Helped |
| Add a frozen LLaMA block to Point-BERT | Point-BERT · ScanObjectNN | accuracy: 87.4 -> 88.0 (BG); 88.1 -> 88.5 (OBJ); 83.1 -> 83.8 (T50) | Helped |
| Add a frozen LLaMA block to Point-BERT | Point-BERT · ModelNet40 | accuracy: 92.67 -> 92.42 (1k); 92.91 -> 92.82 (4k); 93.19 -> 93.56 (8k) | Mixed |
| Add a frozen LLaMA block to METER | METER · Flickr30k Val | IR@1: 49.66 -> 50.22 | Helped |
| Add a frozen LLaMA block to mmTransformer | mmTransformer · Argoverse | ADE: 0.72 -> 0.71 | Helped |
| Replace the LLaMA block with a parameter-matched MLP (ablation) | ViT-S · ImageNet | accuracy: 80.7 -> 80.4 | Hurt |
| Fine-tune instead of freeze the LLaMA transformer (ablation) | ViT-S-LLaMA · ImageNet, 300 epochs | accuracy: 80.7 -> 78.9 | Hurt |
| Fine-tune instead of freeze the LLaMA transformer (ablation) | ViT-S-LLaMA · ImageNet, 100 epochs | accuracy: 75.8 -> 76.8 | Helped |
| Replace pretrained LLaMA weights with a randomly initialized LLM block (ablation) | ViT-S · ImageNet | accuracy: 80.7 -> 76.9 | Hurt |
| Use a frozen OPT-125M block (ablation) | ViT-S · ImageNet | accuracy: 75.25 -> 71.63 | Hurt |
| Use a frozen OPT-6.7B block (ablation) | ViT-S · ImageNet | accuracy: 75.25 -> 76.29 | Helped |
| Insert the single LLaMA block at the head instead of the tail (ablation) | ViT-S · ImageNet | Acc Top 1: 75.84 -> 72.66 | Hurt |
Surprising findings
- Randomly initialized LLM transformer blocks hurt performance, while pre-trained frozen blocks help, suggesting pre-trained weights matter beyond capacity.
- Fine-tuning the LLM block hurts accuracy after 300 epochs due to overfitting, although it helps at 100 epochs.
- OPT blocks at 125M and 350M scales reduce ImageNet accuracy; gains only emerge at 1.3B and above.
Limitations
- Hypothesis does not separately quantify the function of different layers.
- Does not explain how training dynamics let visual token features cooperate with the frozen LLM block.
- Experiments are not scaled to state-of-the-art levels due to compute; not aiming for SOTA.
Experimental setup
Datasets: ImageNet-1k ImageNet-C ImageNet-A ImageNet-SK ImageNet-R ImageNet-S ScanObjectNN ModelNet40 Something-Something-v2 Argoverse VQAv2 Flickr30k
Benchmarks: ImageNet-1k ImageNet-C ImageNet-A ImageNet-SK ImageNet-R ScanObjectNN ModelNet40 Something-Something-v2 Argoverse VQAv2 Flickr30k SQA3D
Metrics: top-1 accuracy top-5 accuracy ADE FDE miss rate mIoU exact match IR@1
Baselines: ViT-S ViT-B ViT-T Point-BERT VideoMAE VectorNet mmTransformer METER SQA3D ScanQA ViT-S-MLP Random LLM
Builds on: ViT DeiT VideoMAE Point-BERT METER
Closest papers by meaning
- Beyond Text: Frozen Large Language Models in Visual Signal Comprehension (CVPR 2024 · similarity 0.91)
- Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers (AAAI 2025 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.90)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.