Research map / Large language models / Language, safety and interpretability

Frozen Language Models Are Gradient Coherence Rectifiers in Vision Transformers

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a frozen LLaMA block inserted into a ViT acts as a gradient coherence rectifier (higher GSNR), and uses it as a training-only auxiliary gradient path that improves vanilla ViT accuracy with no inference cost.

Problem

Explaining why concatenating a frozen pretrained LLM block to a vision transformer improves vision performance, and how to exploit this without adding inference cost.

Approach

Analyzes frozen LLM blocks through layer-wise gradient signal-to-noise ratio (GSNR), finds they increase gradient coherence in preceding ViT blocks, links gain to representation similarity (Mutual-KNN) between the LLM block and adjacent ViT block, and proposes an auxiliary training path through the frozen LLM block that is discarded at inference.

Key results

• Auxiliary training raised ViT-S on CIFAR-100 acc@1 from 71.62 to 73.37 and Caltech-256 from 47.16 to 48.15.
• On ImageNet-1K, ViT-B acc@1 improved from 80.6 to 80.9 with auxiliary training, and ViT-S from 80.1 to 80.3.
• Inserting a LLaMA-7B block at the end of DeiT raised CIFAR-100 accuracy to 74.28 (layer 23) vs 70.65 for layer 0 insertion, with higher Mutual-KNN similarity correlating with larger gains.

Measured effects reported in the paper

Surprising findings

• Inserting the LLM block at the beginning of ViT-S on ImageNet reduced accuracy (71.56, -3.69), and the gains are not monotonic with LLM size for small blocks.
• Larger auxiliary loss weight beyond a point (lambda=0.45) hurt performance, indicating too-strong gradient rectification introduces noise.

Limitations

• Gains on ImageNet are small (about +0.2 to +0.3 points) and gains on SSv2 fine-tuning are not consistent.
• Analysis is limited to ViT-S/ViT-B/ViT-T and LLaMA blocks on a few datasets, with fine-tuned rather than from-scratch SSv2 training.
• Mechanism explanation relies on GSNR and Mutual-KNN correlations without a causal test.

Experimental setup

Datasets: ImageNet-1K Something-Something-v2 CIFAR-100 Caltech-256 BAR

Benchmarks: ImageNet-1K ImageNet-C ImageNet-A ImageNet-SK ImageNet-R Something-Something-v2 CIFAR-100 Caltech-256 BAR

Metrics: top-1 accuracy top-5 accuracy GSNR Mutual-KNN score

Baselines: ViT-S ViT-B ViT-T DeiT

Builds on: Pang et al. 2023 frozen LLM blocks as general-purpose encoders Huh et al. 2024 platonic representation hypothesis Mutual-KNN similarity (Park et al. 2023) GSNR (Liu et al. 2019) DeiT

Closest papers by meaning

• Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.90)
• IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (AAAI 2025 · similarity 0.89)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.88)
• Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.88)