Research map / Vision-language and multimodal / Multimodal LLMs

TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TG-LLaVA, which guides the CLIP/SigLIP vision encoder with text via learnable latent embeddings (TG-FOM mask and TG-DP high-resolution detail perceiver) on top of LLaVA-1.5, improving benchmarks without extra training data.

Problem

Existing VLM improvements focus on the connector and LLM, while the vision encoder's features are computed from the image alone without interaction with the textual instruction, so question-relevant regions may be diluted.

Approach

Use learnable latent queries to parse the instruction and inject the result into visual features via a zero-initialized linear layer (TG-FOM), and use a second latent set to extract instruction-relevant details from high-resolution patches (TG-DP), appending these detail tokens to the visual tokens.

Key results

• With Vicuna-7B and CLIP-L, TG-LLaVA reaches 61.3 on MMBench vs 59.1 for LLaVA-1.5 and 70.6 on ScienceQA vs 69.2, averaging about 1.5% gain over ten benchmarks.
• On the contemporaneous-method comparison table, TG-LLaVA scores MME 1603, MMBench 70.2, MMVet 36.6, GQA 63.4, exceeding LLaVA-1.5 (1531, 67.7, 35.4, 63.3).
• With Qwen2-7B and a 0.5M+10M training setup, TG-LLaVA reports MMBench 74.2, MathVista 52.6, ScienceQA 88.1 and MME 1993, with the authors citing an average 8% gain over ten datasets.

Measured effects reported in the paper

Surprising findings

• Inserting the text-guided module at the head or middle of the vision encoder severely degrades MME (e.g., 711 or 694 vs 1603 at the output), which the authors attribute to unaligned shallow features.
• The FOM module alone lowers ScienceQA (67.5 vs 69.2 baseline) and AI2D (55.1 vs 55.5).

Limitations

• Adds training time (about 10%) and parameters, and processes detail patches sequentially.
• Shallow-layer insertion of text guidance degrades performance, so the design only works at the encoder output.
• Gains are modest on some benchmarks (e.g., MME drops to 1779 for Vicuna-7B and LLaVA-Bench varies), and most results come from the authors' own replication.

Experimental setup

Datasets: LLaVA-1.5 pretraining and instruction tuning data Mini-Gemini dataset (1.2M+1.5M) Ovis training data (10M visual captions subset)

Benchmarks: MMBench MMStar MMMU MathVista OCRBench AI2D HallusionBench LLaVA-Bench ScienceQA MME MMVet GQA

Metrics: accuracy score

Baselines: LLaVA-1.5 Seeing the image TokenPacker DenseConnector MiniGPT4 Qwen-VL VisualGLM PandaGPT mPLUG-Owl2 Emu2-chat Yi-VL ShareGPT-4V

Builds on: LLaVA-1.5 CLIP BLIP-2 Q-former SAM-Adapter

Closest papers by meaning

• TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations (AAAI 2025 · similarity 0.90)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
• Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
• HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models (CVPR 2025 · similarity 0.89)
• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.88)
• LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.88)