Research map / Vision-language and multimodal / Cross-modal learning

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HeRA, which aligns the least MKNN-aligned attention heads of an LLM to a DINOv2 teacher via a contrastive local-neighborhood loss during LLaVA training, improving vision-centric VQA and reducing hallucinations across several LLMs.

Problem

Existing MLLM representation alignment methods regularize a fixed layer of the language backbone to match a vision encoder, ignoring finer-grained structure; it is unclear which internal representation to align and whether enforcing local cross-modal topological alignment improves MLLMs.

Approach

Select individual attention heads by their pre-training MKNN alignment score with a vision teacher and apply a multi-positive contrastive loss that matches the local neighborhood structure of pooled head outputs to the teacher's visual neighborhoods; aligning the least aligned heads gives the largest gains.

Key results

• HeRA on Qwen3-8B raises the Vision-centric average from 55.9 to 59.5 (+3.6), exceeding JARVIS (+2.8) among compared methods.
• On Qwen2.5-3B, aligning the worst-5 heads gives a Vision average of 52.9 vs 50.5 for LLaVA baseline; aligning top-5 heads gives 50.2 (less gain).
• HeRA improves Qwen2.5-14B Vision average by +3.4 (54.9 to 58.3) and reduces CHAIR_s on MSCOCO from 42.6 to 39.0 for that model.

Surprising findings

• Aligning the least aligned attention heads yields larger gains than aligning the best aligned heads or the middle layer.
• Feature-level cosine alignment of visual and text features is ineffective, and DINOv2-L alone is a poor MLLM vision encoder.

Limitations

• Head selection depends on a precomputed MKNN ranking and uses a fixed number of heads (m=5) chosen empirically.
• Gains on some benchmarks are mixed; e.g., Qwen3-4B drops on HallusionBench qAcc and some vision subtasks show regressions.
• Hallucination reduction is not explicitly targeted by the objective and is reported as a side effect.

Experimental setup

Datasets: LLaVA-1.5 558k image-caption pairs LLaVA-Instruct-665k GranD (1,000 samples for MKNN head scoring)

Benchmarks: GQA POPE MME MMBench SEED-Bench ScienceQA MMMU MathVista AI2D ChartQA OCRBench TextVQA

Metrics: accuracy average score over category CHAIR_s CHAIR_i AMBER cover/HalRate/Cog F1 qAcc

Baselines: LLaVA-1.5 training recipe ROSS VIRAL JARVIS CMAR feature-level cosine alignment

Builds on: LLaVA-1.5 Platonic Representation Hypothesis mutual k-nearest neighbor alignment InfoNCE VIRAL ROSS JARVIS CMAR

Closest papers by meaning

• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.90)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.90)
• Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning (NeurIPS 2026 · similarity 0.90)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
• Towards Multimodal Large Language Models with Both Training and Inference Efficiency (ICML 2026 · similarity 0.89)