Research map / Vision-language and multimodal / Visual question answering
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TaxonomiGQA, a text-only taxonomic QA dataset derived from GQA, and compares seven LM–VLM minimal pairs to show VL training mainly improves how taxonomic knowledge is deployed rather than changing the knowledge itself.
Problem
Whether vision-and-language training meaningfully changes the linguistic, specifically taxonomic lexical-conceptual, knowledge of language models, given that prior work finds mostly marginal text-only differences.
Approach
Compare minimal pairs of text-only LMs and their VL-trained counterparts on a text-only taxonomic QA task (TaxonomiGQA) built from GQA scene graphs and WordNet hypernym chains, then separate taxonomic knowledge (direct judgments, representation RSA, embedding similarity) from knowledge deployment (contextual similarity, PCA separability of question representations).
Key results
- Most VLMs outperform their text-only LM counterparts on TaxonomiGQA across metrics, with the Qwen2.5-I vs. Qwen2.5-VL-I pair showing the largest gap, especially on conditional accuracy and HC.
- On TaxoMPs direct taxonomic judgments, most VLM-LM pairs perform similarly; RSA between VLM and LM unembedding spaces is high (≥0.95), indicating shared taxonomic structure.
- VLM question representations are more linearly separable by taxonomic vs. non-taxonomic relations (lower SVM error), supporting the deployment hypothesis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VL training to Llama-3.1 | Llama-3.1 · Rodriguez et al. dataset | accuracy: 0.50 -> 0.68 | Helped |
| Add VL training to Llama-3.1-I | Llama-3.1-I · Rodriguez et al. dataset | accuracy: 0.51 -> 0.50 | Hurt |
| Add VL training to Mistral-v0.2-I | Mistral-v0.2-I · Rodriguez et al. dataset | accuracy: 0.69 -> 0.78 | Helped |
| Add VL training to Qwen2 | Qwen2 · Rodriguez et al. dataset | accuracy: 0.60 -> 0.81 | Helped |
| Add VL training to Qwen2-I | Qwen2-I · Rodriguez et al. dataset | accuracy: 0.84 -> 0.81 | Hurt |
| Add VL training to Qwen2.5-I | Qwen2.5-I · Rodriguez et al. dataset | accuracy: 0.78 -> 0.83 | Helped |
| Add VL training to Vicuna | Vicuna · Rodriguez et al. dataset | accuracy: 0.50 -> 0.74 | Helped |
| Add VL training to language models | seven LM-VLM minimal pairs · TaxonomiGQA | accuracy | Mixed |
| Add VL training to language models | seven LM-VLM minimal pairs · TAXOMPS | overall accuracy | No clear effect |
| Use VL-trained rather than text-only contextual representations | Qwen2.5-I and Qwen2.5-VL-I · TaxonomiGQA | association of contextual similarity with correctness | Helped |
| Use VL-trained question representations | Qwen2.5-I and Qwen2.5-VL-I · TaxonomiGQA | SVM error | Helped |
| Use VL-trained rather than text-only static embeddings | Qwen2.5-I and Qwen2.5-VL-I · TaxonomiGQA taxonomy | hyponym-hypernym similarity difference: 0.04 -> 0.03 | Hurt |
Surprising findings
- VLMs outperform their LM counterparts on a purely text-only QA task despite no visual input at test time.
- Visual similarity of hyponym-hypernym members predicts VLM accuracy significantly (b=0.52, p<.01) but much less for the LM (b=0.23, p=0.18).
Limitations
- Analyses are correlational and do not provide causal evidence linking representations to behavior.
- VL-tuned models see more text data in addition to visual supervision, a confound not separated out.
- SVM separability analysis only captures linearly encoded taxonomic distinctions.
Experimental setup
Datasets: GQA validation split WordNet THINGS Rodriguez et al. property inheritance dataset
Benchmarks: TaxonomiGQA TaxoMPs Rodriguez et al. property inheritance dataset
Metrics: overall accuracy conditional accuracy hierarchical consistency RSA correlation SVM error
Baselines: Llama-3.1-8B Vicuna-7B Mistral-7B-Instruct-v0.2 Qwen2-7B Qwen2-7B-Instruct Qwen2.5-7B-Instruct Llama-3.1-8B-Instruct
Builds on: GQA WordNet causal separability of concepts (Park et al. hierarchical structure method) Rodriguez et al. property inheritance dataset
Closest papers by meaning
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.87)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.87)
- Words or Vision: Do Vision-Language Models Have Blind Faith in Text? (CVPR 2025 · similarity 0.87)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.87)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.