Research map / Vision-language and multimodal / Human and embodied modeling
A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KnoBo, a concept bottleneck model whose concepts are retrieved from medical corpora (PubMed, textbooks, etc.) via RAG-prompted LLMs and whose weights are regularized by LLM-derived sign priors, to improve robustness of medical image classifiers under confounded domain shifts.
Problem
Medical image classifiers (chest X-ray, skin lesion) degrade sharply under domain shifts and spurious confounds (e.g., sex, race, hospital), and existing visual backbones lack an appropriate prior for these domains.
Approach
Build concept bottlenecks from medical documents via iterative retrieval-augmented LLM concept generation, ground concepts with classifiers trained on LLM-annotated image-text pairs, and add a parameter prior that constrains the linear layer's concept-label weight signs to LLM-specified expectations.
Key results
- KnoBo improves OOD accuracy over the strongest end-to-end baseline (ViT-L/14) by 41.8% on X-ray and 22.9% on skin lesion confounded datasets, with domain gap reduced to 30.9 (X-ray) and 14.1 (skin) versus 79.7 and 48.0 for ViT-L/14.
- On unconfounded X-ray datasets KnoBo trails the best black-box Linear Probe by only 0.7% (73.1 vs 73.8 unconfounded accuracy).
- PubMed-derived bottlenecks gave the best X-ray overall score (73.7) and highest concept diversity (0.619) among knowledge sources.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KnoBo instead of LSL | medical image classification · Chest X-ray confounded datasets | OOD accuracy: 55.1 -> 58.8 | Helped |
| use KnoBo instead of Linear Probe | medical image classification · Skin lesion confounded datasets | OOD accuracy: 52.1 -> 70.5 | Helped |
| use KnoBo instead of LSL | medical image classification · Chest X-ray datasets | overall accuracy: 69.0 -> 73.7 | Helped |
| use KnoBo instead of Linear Probe | medical image classification · Skin lesion datasets | overall accuracy: 77.4 -> 78.2 | Helped |
| use PubMed rather than prompted concepts without document retrieval | KnoBo · Chest X-ray datasets | overall accuracy: 72.9 -> 73.7 | Helped |
| use PubMed rather than prompted concepts without document retrieval | KnoBo · Skin lesion datasets | overall accuracy: 77.7 -> 78.2 | Helped |
| add learned concept grounding instead of using prompted CLIP dot-products as concepts (ablation) | KnoBo · Chest X-ray datasets | OOD accuracy: 51.5 -> 58.8 | Helped |
| add learned concept grounding instead of using prompted CLIP dot-products as concepts (ablation) | KnoBo · Skin lesion datasets | overall accuracy: 73.4 -> 78.2 | Helped |
| add the parameter prior (ablation) | KnoBo · Chest X-ray datasets | OOD accuracy: 48.1 -> 58.8 | Helped |
| add the parameter prior (ablation) | KnoBo · Skin lesion datasets | OOD accuracy: 69.1 -> 70.5 | Helped |
| add the parameter prior (ablation) | KnoBo · Chest X-ray datasets | ID accuracy: 91.6 -> 89.7 | Hurt |
| add the parameter prior (ablation) | KnoBo · Skin lesion datasets | ID accuracy: 86.5 -> 86.0 | Hurt |
Surprising findings
- Pixel features outperform features from untrained-then-linear-probed ViT-L/14 and ConvNext-L on X-ray and skin lesion datasets, suggesting deep backbones lack useful image priors for medical data.
- KnoBo is less competitive on skin lesion unconfounded datasets, attributed to lack of large-scale pretraining data for concept grounding.
Limitations
- Assumes availability of medical multimodal paired datasets, limiting use for rare conditions.
- Groundability of text-derived concepts to images is lower than relevance, showing difficulty deriving visual concepts from text-only knowledge.
- Skin lesion performance is weaker on unconfounded data due to limited large-scale pretraining data.
Experimental setup
Datasets: MIMIC-CXR ISIC NIH-CXR CheXpert Pneumonia COVID-QU Open-i VinDr-CXR HAM10000 BCN20000 PAD-UFES-20 Melanoma
Benchmarks: NIH-sex NIH-age NIH-pos CheXpert-race NIH-CheXpert ISIC-sex ISIC-age ISIC-site ISIC-color ISIC-hospital Pneumonia COVID-QU
Metrics: accuracy in-domain accuracy out-of-domain accuracy domain gap average accuracy concept diversity relevance groundability
Baselines: ViT-L/14 DenseNet121 Linear Probe Language-shaped Learning Post-hoc CBM LaBo
Builds on: Concept Bottleneck Models LaBo Post-hoc CBM Language-shaped Learning CLIP MedRAG
Closest papers by meaning
- Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.86)
- Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation (CVPR 2024 · similarity 0.86)
- LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologies (AAAI 2025 · similarity 0.86)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.85)
- Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models (NeurIPS 2026 · similarity 0.85)
- MedSG-Bench: A Benchmark for Medical Image Sequences Grounding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.