Research map / Vision-language and multimodal / Cross-modal learning
Modality-Agnostic Structural Image Representation Learning for Deformable Multi-Modality Medical Image Registration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MASR-Net, a self-supervised network learning modality-agnostic deep structural image representations via deep neighbourhood self-similarity and anatomy-aware contrastive learning, used to make multimodal deformable registration a monomodal problem.
Problem
Deformable multi-modality medical image registration needs dense anatomical correspondence across modalities, but statistical similarity measures (MI) are sensitive to noise and local structural descriptors (MIND) lack discriminability; supervised learning needs pre-aligned or segmented training dat
Approach
Learn deep structural image representations with a CNN plus a deep neighbourhood self-similarity (DNS) module, trained contrastively with non-linear Bezier intensity transforms and random intensity inversion so that features are contrast-invariant and anatomy-discriminative, without delineations or aligned pairs; then register with cosine-similarity loss in LapIRN or instance optimization.
Key results
- On liver multiphase CT, IO+DNS achieves average DSC 87.78 across three directions versus 84.82 for IO+MIND and 87.46 for DEEDs (MIND).
- On Abdomen MR-CT, IO+DNS reaches DSC 85.61 versus 83.53 for DEEDs (MIND), with runtime 4.52 s versus 165.21 s.
- On brain MR T1w-T2w, IO+DNS raises initial DSC from 53.90 to 62.66 and 61.91, comparable to NiftyReg and ANTs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DNS instead of MIND | IO · Abdomen MR-CT | DSC | Helped |
| Use DNS instead of MIND | LapIRN · Liver multiphase CT, average across three tasks | DSC | Helped |
| Use DNS instead of MIND | IO · Liver multiphase CT, average across three tasks | DSC: +4.16 DSC | Helped |
| Add DNS to randomly initialized backbone features (ablation) | MASR-Net · Liver multiphase CT, Pre-contrast ← Venous & Arterial | DSC: +11.12 DSC | Helped |
| Add nonlinear intensity augmentation and contrastive learning (ablation) | MASR-Net with DNS · Liver multiphase CT, Pre-contrast ← Venous & Arterial | DSC: +0.63 | Helped |
| Add Gaussian smoothing (ablation) | MASR-Net · Liver multiphase CT, Pre-contrast ← Venous & Arterial | DSC: +0.22 | Helped |
| Use randomly initialized backbone features without DNS (ablation) | MASR-Net · Liver multiphase CT, Pre-contrast ← Venous & Arterial | DSC | Hurt |
Surprising findings
- Random-initialized CNN features reduce DSC below the initial unregistered baseline (75.38 vs 81.77), showing random features alone are insufficient for multimodal registration.
- Mutual information-based ANTs and NiftyReg fail on abdomen MR-CT (DSC 39.43 and 48.60), indicating MI cannot handle large-deformation multimodal registration.
Limitations
- Iterative optimization-based methods only evaluated on abdomen MR-CT and brain tasks due to limited training data.
- Brain MR T1w-T2w gains are on-par rather than superior to conventional methods (NiftyReg, ANTs).
Experimental setup
Datasets: in-house liver multiphase CT (1966 patients) Learn2Reg 2021 Abdomen MR-CT BraTS18 ISeg19
Benchmarks: in-house liver multiphase CT registration Learn2Reg 2021 Abdomen MR-CT BraTS18/ISeg19 brain MR T1w-T2w
Metrics: DSC HD95 percentage of Jacobian determinant < 0 runtime
Baselines: ANTs NiftyReg DEEDs VoxelMorph LapIRN MIND NMI MI
Builds on: MIND LapIRN PatchNCE Bezier curve intensity transformation
Closest papers by meaning
- Multi-modal Vision Pre-training for Medical Image Analysis (CVPR 2025 · similarity 0.86)
- Modality-agnostic Domain Generalizable Medical Image Segmentation by Multi-Frequency in Multi-Scale Attention (CVPR 2024 · similarity 0.85)
- Deep Learning in Medical Image Registration: Magic or Mirage? (NeurIPS 2024 · similarity 0.85)
- Correlation-aware Coarse-to-fine MLPs for Deformable Medical Image Registration (CVPR 2024 · similarity 0.84)
- UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation (NeurIPS 2025 · similarity 0.84)
- CARL: A Framework for Equivariant Image Registration (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.