Research map / Vision-language and multimodal / Multimodal LLMs

S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes S2D-Align, a three-stage shallow-to-deep SFT curriculum (image-report, reference report, RadGraph-derived key phrases) with a shared memory-based adapter (SMA) for radiology report generation with Vicuna-7B and a frozen Rad-DINO encoder.

Problem

Standard SFT of multimodal LLMs for radiology report generation performs only instance-level image-report alignment, which fails to capture anatomically grounded correspondences between findings and locations, hurting factual correctness of templated reports.

Approach

Progressive Anatomical Grounding (PAG) trains in three stages that add auxiliary signals of increasing granularity (reference report, then RadGraph-derived key phrases), linked by a Shallow-to-Deep Memory Adapter whose shared learnable memory queries pool features across stages; auxiliary signals are dropped at inference.

Key results

• On MIMIC-CXR, S2D-Align reports CE F1 of 0.608 and BLEU-4 of 0.149, the best among compared methods (LLM-RG4 CE F1 0.588).
• On IU X-Ray, S2D-Align (7B) reports BLEU-1 0.512, BLEU-4 0.195 and ROUGE-L 0.407, above R2-LLM (14.2B).
• Ablation: single-stage S1-only training yields CE F1 0.449 versus 0.608 for the full three-stage curriculum.

Surprising findings

• Reversing the stage order (S1→S3→S2) gives a higher CE F1 (0.527) than joint training of all stages (0.470) but lower BLEU-4 than the full model.
• Adding a Q-Former on top of an MLP lowers CE F1 (0.394) relative to the MLP alone (0.473).

Limitations

• Relies on RadGraph annotations to extract key phrases, so training samples lacking annotations are excluded.
• Reference reports require longitudinal patient studies, which are available mainly in MIMIC-CXR.

Experimental setup

Datasets: MIMIC-CXR IU X-Ray RadGraph

Benchmarks: MIMIC-CXR IU X-Ray

Metrics: BLEU-1 BLEU-4 ROUGE-L CE precision CE recall CE F1

Baselines: R2Gen R2GenCMN R2GenGPT LLM-RG4 XrayGPT EKAGen R2-LLM PromptMRG

Builds on: LLM-RG4 Q-Former RadGraph LoRA

Closest papers by meaning

• DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.89)
• Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · similarity 0.89)
• Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.89)
• CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation (CVPR 2026 · similarity 0.87)
• DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · similarity 0.87)
• LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.87)