Research map / Vision-language and multimodal / Human and embodied modeling
Language Model as Visual Explainer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Supplementary material for LVX, which explains vision models via LLM-built attribute trees with text-to-image support sets, adding chest X-ray, self-supervised model, raw result, and metric/setup details.
Problem
Interpreting the internal decision rationale of vision classifiers with tree-structured language explanations, without model training.
Approach
Builds category-level attribute trees by prompting an LLM (ChatGPT), grounds nodes with support images from a text-to-image API, refines trees iteratively by grow/prune based on visit counts, and explains a prediction by matching a subtree of the predicted class's tree to the image features.
Key results
- On Chestx-ray14, the LVX-calibrated DenseNet-121 raises average AUC from 0.812 to 0.821 across 14 findings; Hernia rises from 0.914 to 0.936.
- On ImageNet, ResNet50 explanation TED drops to 9.09 with LVX versus 9.68 for random and 33.58 for constant baseline.
- Among self-supervised ResNet50 models, DINO and SwAV reach 75.3% top-1 with TED 9.14 and 9.15; DINO ViT-S/16 reaches 77.0% top-1 with TED 8.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LVX instead of Random explanations | VGG13 · CIFAR-10 | TED: 9.21 -> 8.21 | Helped |
| Use LVX instead of Random explanations | VGG13 · CIFAR-10 | MCS: 28.98 -> 32.31 | Helped |
| Use LVX instead of Random explanations | VGG13 · CIFAR-10 | Tree Kernel: 59.49 -> 63.43 | Helped |
| Use LVX instead of Constant explanations | VGG13 · CIFAR-10 | TED: 32.97 -> 8.21 | Helped |
| Use LVX instead of Constant explanations | VGG13 · CIFAR-10 | MCS: 18.77 -> 32.31 | Helped |
| Use LVX instead of Constant explanations | VGG13 · CIFAR-10 | Tree Kernel: 58.41 -> 63.43 | Helped |
| Use LVX instead of Random explanations | ResNet20 · CIFAR-100 | TED: 9.61 -> 8.96 | Helped |
| Use LVX instead of Random explanations | ResNet20 · CIFAR-100 | MCS: 22.65 -> 24.89 | Helped |
| Use LVX instead of Random explanations | ResNet20 · CIFAR-100 | Tree Kernel: 45.52 -> 47.70 | Helped |
| Use LVX instead of Constant explanations | ResNet20 · CIFAR-100 | TED: 28.77 -> 8.96 | Helped |
| Use LVX instead of Constant explanations | ResNet20 · CIFAR-100 | MCS: 17.60 -> 24.89 | Helped |
| Use LVX instead of Constant explanations | ResNet20 · CIFAR-100 | Tree Kernel: 46.96 -> 47.70 | Helped |
Surprising findings
- DINO ViT-S/16 self-supervised model gives better attribute explanation than the supervised ViT-S/16 despite slightly lower accuracy.
- Chest X-ray calibration with LVX fine-tuning slightly decreased AUC for some findings (e.g., Fibrosis 0.832 to 0.830, Pneumonia 0.721 to 0.719).
Limitations
- Support dataset may contain out-of-distribution samples and bias from text-to-image retrieval.
- Dataset withheld for privacy concerns until acceptance; no direct baseline for explanation exists so only naive Random and Constant baselines are used.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Chestx-ray14
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Chestx-ray14
Metrics: Tree Edit Distance Maximum Common Subgraph Tree Kernel AUC
Baselines: Random Constant
Builds on: CLIP MedCLIP
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- A Vision Check-up for Language Models (CVPR 2024 · similarity 0.90)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.90)
- Towards Interpreting Visual Information Processing in Vision-Language Models (ICLR 2025 · similarity 0.90)
- Language-Informed Visual Concept Learning (ICLR 2024 · similarity 0.89)
- PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.