Research map / Vision-language and multimodal / Human and embodied modeling
A Concept-Based Explainability Framework for Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoX-LMM, a Semi-NMF dictionary-learning method that decomposes a large multimodal model's token representations into concepts grounded visually (top-activating images) and textually (decoded vocabulary words).
Problem
Interpreting the internal token representations of large multimodal models (LMMs), which combine visual encoders with LLMs, remains largely unexplored.
Approach
Learn a concept dictionary by decomposing a matrix of residual-stream representations of a target token using Semi-NMF (non-negative activations, sparsity), then ground each concept visually via maximally activating images and textually by unembedding it through the LM head to decode words.
Key results
- Semi-NMF achieves top-1 CLIPScore of 0.610 on 'Dog' vs 0.519 for Rnd-Words and 0.546 for Simple, with GT-captions at 0.783.
- Semi-NMF concept overlap is 0.086 for 'Dog' and 0.177 for 'Bus', far lower than Simple (0.371, 0.622) and K-Means (0.501, 0.487), though PCA has lower overlap (0.004, 0.002).
- CLIPScore grounding improves notably between layers 20 and 25, with early layers no better than random words.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Semi-NMF rather than random-word grounding (ablation) | CoX-LMM on DePALM · COCO Dog | CLIPScore top-1: 0.519 $\pm$ 0.05 -> 0.610 $\pm$ 0.09 | Helped |
| Use Semi-NMF rather than random-image representations (ablation) | CoX-LMM on DePALM · COCO Dog | CLIPScore top-1: 0.425 $\pm$ 0.06 -> 0.610 $\pm$ 0.09 | Helped |
| Use Semi-NMF rather than the Simple baseline (ablation) | CoX-LMM on DePALM · COCO Bus | CLIPScore top-1: 0.667 $\pm$ 0.06 -> 0.634 $\pm$ 0.08 | Hurt |
| Use Semi-NMF rather than the Simple baseline (ablation) | CoX-LMM on DePALM · COCO Train | BERTScore top-1: 0.392 $\pm$ 0.07 -> 0.378 $\pm$ 0.07 | Hurt |
| Use Semi-NMF rather than PCA (ablation) | CoX-LMM concept dictionary · COCO Dog, Bus, Train, Cat | concept overlap: Dog: 0.004 -> 0.086; Bus: 0.002 -> 0.177; Train: 0.015 -> 0.107; Cat: 0.000 -> 0.146 | Mixed |
| Increase dictionary size K (ablation) | CoX-LMM · COCO Dog and Cat | CLIPScore and concept overlap | Mixed |
| Use deeper LLM layers (around L=20 to L=25 and beyond) rather than early layers (ablation) | CoX-LMM multimodal grounding · COCO Dog and Cat | grounding CLIPScore | Helped |
| Replace the CLIP visual encoder with an ImageNet-trained ViT-L encoder (ablation) | DePALM with Semi-NMF · COCO Dog | CLIPScore top-1: 0.610 $\pm$ 0.09 -> 0.611 $\pm$ 0.09 | No clear effect |
| Replace the CLIP visual encoder with a MAE-trained ViT-L encoder (ablation) | DePALM with Semi-NMF · COCO Dog | CLIPScore top-1: 0.610 $\pm$ 0.09 -> 0.602 $\pm$ 0.07 | No clear effect |
| Use Semi-NMF rather than the Simple baseline (ablation) | CoX-LMM on LLaVA-v1.5 · COCO Dog | BERTScore top-1: 0.331 $\pm$ 0.07 -> 0.305 $\pm$ 0.07 | Hurt |
Surprising findings
- Replacing the CLIP visual encoder with ImageNet-trained or MAE ViT-L encoders gives similar test CLIPScores, suggesting LLM processing drives the multimodal concept structure.
- PCA yields almost no concept overlap but its grounded words are less coherent with images.
Limitations
- Analysis is limited to a single target token at a time, and the method is evaluated on a small set of tokens.
- Evaluation relies on CLIPScore and BERTScore proxies rather than human judgments; BERTScore required LLaMA-generated phrases.
- Polysemanticity remains in some concepts (e.g., 'Bull dog' concept specificity 50%).
Experimental setup
Datasets: COCO ImageNet
Benchmarks: COCO Karpathy test split
Metrics: CLIPScore BERTScore F1 concept overlap t-test p-values
Baselines: Rnd-Words Noise-Imgs Simple PCA K-Means GT-captions
Builds on: CRAFT Concept activation vectors (TCAV) ACE Logit lens
Closest papers by meaning
- Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data (ICLR 2025 · similarity 0.90)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
- Matryoshka Multimodal Models (ICLR 2025 · similarity 0.89)
- Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation (AAAI 2025 · similarity 0.89)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.88)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.