Research map / Video / Video-language understanding
MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MCAT, a multi-tier class-aware spatio-temporal transformer with visual-query conditioning and specialized contrastive and uncertainty localization losses, to localize standard fetal ultrasound anatomy clips from a visual query.
Problem
Localizing standard-plane anatomy clips in fetal ultrasound sweep videos given a visual query, where neighboring frames are highly similar and annotations are noisy.
Approach
Query-guided cross-attention fusion at multiple backbone tiers, class-specific learnable temporal tokens (only the query's class token is updated) to cut token count by 96%, plus a multi-tier dual-anchor contrastive loss and a Gaussian KL-based temporal uncertainty localization loss.
Key results
- MCAT reaches 34.10 mtIoU on the heart sweep data vs 24.05 for VQLOC, about 10% better.
- On PULSE data MCAT reaches 30.63 mtIoU vs 17.89 for ResNet3D, a 12.74-point gain.
- On Ego4D VQ-VCL MCAT reaches 43.94 mtIoU vs 38.59 for TubeDETR, a 5.35-point gain, using 96% fewer tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MCAT instead of VQLOC | VQ-VCL · Heart Sweep Data | mtIoU: 24.05 -> 34.1 | Helped |
| Use MCAT instead of ResNet 3D | VQ-VCL · PULSE Data | mtIoU: 17.89 -> 30.63 | Helped |
| Use MCAT instead of TubeDETR | VQ-VCL · Ego4D VQ-VCL Dataset | mtIoU: 38.59 -> 43.94 | Helped |
| Add Tier 2 features to Tier 1 (ablation) | MCAT · Heart Sweep Data | mtIoU: 28.23 -> 31.33 | Helped |
| Add Tier 3 features to Tiers 1 and 2 (ablation) | MCAT · Heart Sweep Data | mtIoU: 31.33 -> 34.10 | Helped |
| Add Temporal Uncertainty Aware Localization (URL) loss to Cross-Entropy (ablation) | MCAT · Heart Sweep Data | mtIoU: 15.93 -> 29.64 | Helped |
| Add Multi-Tier Dual Anchor Contrastive (MTDA) loss to URL loss (ablation) | MCAT · Heart Sweep Data | mtIoU: 29.64 -> 34.10 | Helped |
| Use parallel instead of sequential fusion (ablation) | MCAT · Heart Sweep Data | mtIoU: 31.77 -> 34.10 | Helped |
| Use cross-attention instead of concatenation and self-attention for video-query fusion (ablation) | MCAT · Heart Sweep Data | mtIoU: 14.47 -> 34.10 | Helped |
| Use class-specific tokens instead of a generic embedding (ablation) | MCAT · Heart Sweep Data | mtIoU: 31.00 -> 34.10 | Helped |
| Add tier-specific embeddings (ablation) | MCAT · Heart Sweep Data | mtIoU: 30.58 -> 34.10 | Helped |
Surprising findings
- Human experts agreed on standard frame selection with only a 66% kappa score between two fetal cardiologists.
Limitations
- Ego4D VQ-VCL dataset and code release is only planned, not confirmed.
- Clinical datasets are small (200 training videos) and private, limiting reproducibility.
Experimental setup
Datasets: CAIFE fetal heart sweep videos PULSE fetal anomaly scan videos Ego4D
Benchmarks: CAIFE heart sweep data PULSE data Ego4D VQ-VCL
Metrics: mtIoU R@0.1 R@0.3 R@0.5 R@0.7
Baselines: CS Sup CNN TubeDETR MomentDETR ResNet3D VQLOC
Builds on: TubeDETR MomentDETR VQLOC cross-attention transformer
Closest papers by meaning
- FETAL-GAUGE: A BENCHMARK FOR ASSESSING VISION-LANGUAGE MODELS IN FETAL ULTRASOUND (ICLR 2026 · similarity 0.88)
- U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.85)
- F$^2$-Assist: Multi-Phase Fetal Growth Forecast and Report Generation from Ultrasound Examination (CVPR 2026 · similarity 0.85)
- Unsupervised Domain Adaptation for Anatomical Structure Detection in Ultrasound Images (ICML 2024 · similarity 0.84)
- M3-UDA: A New Benchmark for Unsupervised Domain Adaptive Fetal Cardiac Structure Detection (CVPR 2024 · similarity 0.84)
- Self-supervised Learning of Echocardiographic Video Representations via Online Cluster Distillation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.