Research map / Vision-language and multimodal / Human and embodied modeling
Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how vision-encoder (CLIP) knowledge affects LVLM landmark recognition via a new multi-granularity landmark dataset (MGLD), then proposes EECA, which supervises adapter visual tokens with entity-aware contrastive and hierarchical classification losses.
Problem
LVLMs built from separately pretrained vision encoders and LLMs suffer "cognitive misalignment": visual features the VE produces do not match the LLM's recognition space, so they fail at landmark recognition even when the LLM knows the landmark from text.
Approach
Categorize training images as VE-Known or VE-Unknown using CLIP similarity and rank metrics, show VE-Known data helps and VE-Unknown hurts, then train the adapter's visual tokens with entity-aware contrastive loss over high-resolution tokens plus a hierarchical classification loss so they mimic VE-Known, discriminative representations.
Key results
- Selecting VE-Known data (HDS) at 25k raised accuracy from 8.68% baseline to 49.3% relative gain (+9.2 over BRS 40.1); VE-Unknown LCS dropped to 23.0 (-17.1 vs BRS).
- EECA with entity prompts (training + inference) reached 19.52 Strongly Known in Table 2 and EECA total accuracy rose from 8.68 to 15.52 in the ablation, with Le and Lh giving the main gains.
- EECA with 25k data matched the performance of the 125k reference dataset and achieved best results with 50k data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select VE-Known HDS data instead of BRS reference data | LLaVA landmark recognition · Landmark recognition, 25k data | accuracy increase over baseline (%): 40.1 -> 49.3 (+9.2) | Helped |
| Select VE-Unknown LCS data instead of BRS reference data | LLaVA landmark recognition · Landmark recognition, 25k data | accuracy increase over baseline (%): 40.1 -> 23.0 (–17.1) | Hurt |
| Select VE-Known HDS data instead of BRS reference data | LLaVA landmark recognition · Landmark recognition, 50k data | accuracy increase over baseline (%): 56.2 -> 64.1 (+7.9) | Helped |
| Select VE-Unknown LCS data instead of BRS reference data | LLaVA landmark recognition · Landmark recognition, 50k data | accuracy increase over baseline (%): 56.2 -> 28.1 (–28.1) | Hurt |
| Add HSS-50k data (ablation) | LLaVA-1.5 landmark recognition · HSS-50k | accuracy: 8.68 -> 13.92 (+5.24) | Helped |
| Add a high-resolution branch without targeted supervision (ablation) | LLaVA-1.5 with HSS-50k · HSS-50k | accuracy: 13.92 -> 13.88 (-0.04) | Hurt |
| Add entity-aware contrastive loss (ablation) | LLaVA-1.5 with HSS-50k and HR branch · HSS-50k | accuracy: 13.88 -> 14.4 (+0.52) | Helped |
| Add hierarchical classification loss (ablation) | LLaVA-1.5 with HSS-50k and entity-aware contrastive loss · HSS-50k | accuracy: 14.4 -> 15.52 (+1.12) | Helped |
| Add a high-resolution branch (ablation) | LLaVA-1.5 landmark recognition · HDS-25k | accuracy: 13.00 -> 13.60 (+0.60) | Helped |
| Add entity-aware contrastive loss (ablation) | LLaVA-1.5 with HDS-25k and HR branch · HDS-25k | accuracy: 13.60 -> 14.00 (+0.40) | Helped |
| Add hierarchical classification loss (ablation) | LLaVA-1.5 with HDS-25k and entity-aware contrastive loss · HDS-25k | accuracy: 14.00 -> 14.40 (+0.40) | Helped |
| Add hierarchical classification loss (ablation) | LLaVA-1.5 with HSS-25k and entity-aware contrastive loss · HSS-25k | accuracy: 14.40 -> 13.84 | Hurt |
Surprising findings
- Data quality (VE-Known selection) mattered more than data volume: 25k VE-Known samples beat larger random reference sets.
- Adding a high-resolution branch alone did not improve accuracy without targeted supervision (13.88 vs 13.92 in ablation).
Limitations
- Relies on entity and hierarchical labels for supervision, limiting use on tasks without annotations.
- Evaluated mainly on landmark recognition; generality to other tasks not shown.
- Results rely on GPT-4o-based response classification and GPT-4o-generated annotations.
Experimental setup
Datasets: GLDv2 MGLD
Benchmarks: MGLD landmark test split
Metrics: accuracy (proportion of Strongly Known and Known responses) Strongly Known rate Known rate
Baselines: LLaVA-1.5 GPT-4o Qwen2-VL BRS random selection Mini-Gemini
Builds on: LLaVA-1.5 CLIP MiniGemini Perceiver resampler
Closest papers by meaning
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.91)
- Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding (ICLR 2026 · similarity 0.90)
- Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.89)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.89)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.