Research map / Vision-language and multimodal / Human and embodied modeling
AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AHIVE, a hierarchical, anatomy-aware visual encoder trained with CLIP on multi-level diagnostic descriptions derived from Chest ImaGenome, enabling clinically accurate and user-interactive radiology report retrieval on MIMIC-CXR.
Problem
Radiology report retrieval from chest X-rays often lacks clinical accuracy because anatomical and diagnostic details are not explicitly modeled, and existing retrieval systems offer limited interactive user refinement.
Approach
A multi-layer vision encoder that conditions each diagnostic level on anatomy-aware spatial attention and on the visual and language context of preceding levels (anatomy -> anatomy property -> abnormality -> free-text report), so user edits at one level propagate to subsequent retrieval levels.
Key results
- CLIP(SapBERT)+AHIVE reaches CE(11/5) 0.604, CE(11) 0.678 and CE(13+NL) 0.685, the best among compared methods on these clinical efficacy metrics.
- AHIVE improves RadRQI-F1(11) to 0.407 from 0.242 for CLIP(SapBERT), a reported 68.18% gain.
- AHIVE underperforms CLIP(SapBERT) on RadRQI-F1 Hits (30 vs 43), attributed to the limited set of abnormalities modeled.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AHIVE hierarchical anatomy-aware encoding | CLIP(SapBERT) · MIMIC-CXR test | CE(11): 0.642 -> 0.678 | Helped |
| add AHIVE hierarchical anatomy-aware encoding | CLIP(SapBERT) · MIMIC-CXR test | RadRQI-F1(11): 0.242 -> 0.407 | Helped |
| add AHIVE hierarchical anatomy-aware encoding | CLIP(SapBERT) · MIMIC-CXR test | RadRQI-F1 Hits: 43 -> 30 | Hurt |
| add AHIVE hierarchical anatomy-aware encoding | CLIP(SapBERT) · MIMIC-CXR test | BLEU: 0.202 -> 0.131 | Hurt |
| use N=3 anatomical parts instead of N=5 (ablation) | AHIVE · MIMIC-CXR test | CE(K): 0.678 -> 0.663 | Hurt |
| use N=9 anatomical parts instead of N=5 (ablation) | AHIVE · MIMIC-CXR test | CE(K/N): 0.604 -> 0.528 | Hurt |
| use K=5 abnormalities instead of K=11 (ablation) | AHIVE · MIMIC-CXR test | CE(K): 0.678 -> 0.589 | Hurt |
| use K=16 abnormalities instead of K=11 (ablation) | AHIVE · MIMIC-CXR test | RadRQI-F1(Top-50): 0.327 -> 0.355 | Helped |
| remove language context at layer 2 (ablation) | AHIVE · MIMIC-CXR test | CE(11/5): 4.0 | Hurt |
| remove visual context at layer 2 (ablation) | AHIVE · MIMIC-CXR test | CE(11/5): 0.2 | No clear effect |
| remove visual and language contexts at layer 2 (ablation) | AHIVE · MIMIC-CXR test | CE(11/5): 7.3* | Hurt |
| remove language context at layer 3 (ablation) | AHIVE · MIMIC-CXR test | CE(11/5): 4.2 | Hurt |
Surprising findings
- AHIVE underperforms the CLIP(SapBERT) baseline on the RadRQI-F1 Hits metric, which the authors attribute to the limited set of pre-defined abnormalities.
Limitations
- Manual template design is needed to extract hierarchical diagnostic descriptions from reports, which may be suboptimal.
- The multi-head attention retrieval favors reports with more visually relevant sentences and may discount reports with prior-study comparisons or follow-up recommendations.
- Only a limited set of abnormalities (11 or 16) is modeled, reducing coverage of the dataset.
Experimental setup
Datasets: MIMIC-CXR Chest ImaGenome
Benchmarks: MIMIC-CXR
Metrics: CE F1 (11/5) CE F1 (11) CE F1 (13+NL) RadRQI-F1 RadRQI-F1 Hits BLEU ROUGE-L CIDEr
Baselines: CLIP(Vanilla) CLIP(SapBERT/Frozen) CLIP(SapBERT) BiomedCLIP MedCLIP X-REM CXR-RePaiR
Builds on: CLIP SapBERT CXR-RePaiR Chest ImaGenome
Closest papers by meaning
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.87)
- Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation (CVPR 2024 · similarity 0.87)
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.86)
- DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.86)
- CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation (CVPR 2026 · similarity 0.86)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.