Research map / Label-efficient and robust learning / Recognition and long-tail
Interactive Medical Image Analysis with Concept-based Similarity Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CSR, an interpretable-by-design medical image classifier that grounds patch-level concept prototypes on image regions and supports doctor interaction at train and test time (spatial boxes, concept rejection).
Problem
Concept-based models give only image-level concept explanations, while prototype-based models give patch-level explanations that need post-hoc work to map to concepts; medical diagnosis needs both interpretability and doctor intervention.
Approach
Learn per-concept patch prototypes via soft-selected local concept vectors and a multi-prototype contrastive loss in a projected space, then classify from max cosine similarities to these prototypes, enabling spatial importance-map and concept-rejection interactions at test time.
Key results
- CSR reaches 94.4 macro F1 on TBX11K, versus 94.1 for the best ProtoPNet and 94.0 for PIP-Net, with explanation size 14 (CBM-like).
- On TBX11K Tuberculosis pointing game, CSR hits 60.9% versus 19.5% for PIP-Net; refined atlas reaches 79.5%.
- Test-time interaction raises macro F1 from 94.4 to 95.1 (+0.7%) with medical student concept plus spatial interaction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CSR instead of ProtoPNet | TBX11K diagnosis · TBX11K | Macro F1-score: 94.1 -> 94.4 | Helped |
| Use CSR instead of CBM (joint.) | VinDr-CXR diagnosis · VinDr-CXR | Macro F1-score: 50.1 -> 54.6 | Helped |
| Use CSR instead of PIP-Net | ISIC diagnosis · ISIC | Macro F1-score: 69.9 -> 71.5 | Helped |
| Refine the concept atlas with doctor review (ablation) | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 94.0 | Hurt |
| Refine the concept atlas with doctor review (ablation) | CSR explanations on TBX11K, Tuberculosis class · TBX11K, Tuberculosis class | Pointing Game hit rate: 60.9 -> 79.5 | Helped |
| Use CSR instead of PIP-Net explanations | TBX11K, Tuberculosis class · TBX11K, Tuberculosis class | Pointing Game hit rate: 19.5 -> 60.9 | Helped |
| Add non-expert spatial interaction | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 94.5 | Helped |
| Add oracle spatial interaction using bounding-box ground truth | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 94.9 | Helped |
| Add M.D. student spatial interaction | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 94.7 | Helped |
| Add M.D. student concept interaction | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 95.1 | Helped |
| Add M.D. student spatial and concept interaction | CSR on TBX11K · TBX11K | Macro F1-score: 94.4 -> 94.9 | Helped |
Surprising findings
- The doctor-refined concept atlas gave a higher pointing game hit rate (79.5%) but slightly lower F1 (94.0), which the authors attribute to shortcuts in the original model.
- Non-expert users with spatial-only interaction gained only +0.1% F1, versus +0.3% to +0.7% for medical student interaction.
Limitations
- Concept-based design relies on concept annotations (findings) available in the datasets.
- Interaction gains are measured with only one medical student and one non-expert, on one dataset.
- Authors leave multimodal foundation-model concepts to future work.
Experimental setup
Datasets: TBX11K VinDr-CXR ISIC
Benchmarks: TBX11K VinDr-CXR ISIC
Metrics: macro F1-score pointing game hit rate explanation size
Baselines: CBM ProtoPNet ProtoPool ProtoTree PIP-Net
Builds on: ProtoPNet Concept Bottleneck Models PIP-Net ProtoTree ProtoPool
Closest papers by meaning
- GRAPE: Graph-Augmented Prototype Explanations for Interactive Medical Image Diagnosis (NeurIPS 2026 · similarity 0.88)
- MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment (AAAI 2024 · similarity 0.86)
- Towards Multi-dimensional Explanation Alignment for Medical Classification (NeurIPS 2024 · similarity 0.85)
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.85)
- Medical thinking with multiple images (ICLR 2026 · similarity 0.84)
- Symb-xMIL: Symbolic Explanations for Multiple Instance Learning in Digital Pathology (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.