Research map / Vision-language and multimodal / Multimodal LLMs

Towards Understanding How Knowledge Evolves in Large Vision-Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how multimodal knowledge evolves across layers of LLaVA-1.5's Vicuna backbone using early-exit token probabilities, JS divergence of layer distributions, and t-SNE of hidden features, identifying critical and mutation layers.

Problem

Understanding how visual features are converted into natural language inside LVLMs, and how hallucinations arise from internal knowledge evolution, remains poorly understood.

Approach

Applies early exit (logit lens style decoding of each intermediate layer), JS divergence between adjacent layer distributions, and t-SNE on hidden features to trace knowledge evolution in three stages (rapid evolution, stabilization, mutation); uses layer skipping to test the role of these stages.

Key results

• Token probabilities stay near zero in shallow layers and rise sharply around layer 20 (critical layers), with some hallucinated tokens mutating in deeper layers.
• JS divergence between adjacent layers drops sharply around layer 18 and stays near zero through the output, indicating a stabilization stage.
• Skipping from critical to mutation layers (skip.1) leaves output semantically close to original; skipping only mutation layers (skip.2) corrects some hallucinations (e.g., 'standing' to 'playing').

Surprising findings

• Skipping mutation layers can correct some hallucinated tokens while preserving most of the original meaning.
• The authors claim the observed phenomena are unique to LVLMs and not seen in text-only LLMs, though the LLM comparison is only a single qualitative example.

Limitations

• Analysis is restricted to LLaVA-1.5 with a Vicuna backbone and a small number of examples and images.
• Hallucination claims and compression/finetuning implications are largely qualitative and not systematically validated.
• The LLM comparison uses a single qualitative example and mislabels LLaMA-1.5.

Experimental setup

Datasets: AMBER MSCOCO

Benchmarks: AMBER MSCOCO

Metrics: token probability per layer Jensen-Shannon divergence hallucination rate (qualitative)

Builds on: early exit LLaVA-1.5

Closest papers by meaning

• Bias Amplification in Language Model Evolution: An Iterated Learning Perspective (NeurIPS 2024 · similarity 0.87)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
• Decouple to Generalize: Context-First Self-Evolving Learning for Data-Scarce Vision-Language Reasoning (CVPR 2026 · similarity 0.86)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.86)
• MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.86)
• Diving into Self-Evolving Training for Multimodal Reasoning (ICML 2025 · similarity 0.86)