Research map / Label-efficient and robust learning / Recognition and long-tail
Novel Class Discovery in Chest X-rays via Paired Images and Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-branch multi-modal novel class discovery method for chest X-rays that weights image and text pseudo-labels by intra-modal structural consistency, evaluated on MIMIC-CXR-JPG splits.
Problem
Novel class discovery for medical images (chest X-rays) is unexplored, and image-only pseudo-labels are inaccurate on CXR data; paired text is available during training but not at test time.
Approach
Quantify pseudo-label quality in each modality by the consistency between embedding-similarity structure and pseudo-label-similarity structure (JS divergence), then fuse the two modalities' pseudo-labels with these consistency scores as weights to supervise both branches, with image-only inference at test time.
Key results
- On task-aware protocol, the method reaches 56.9 / 53.7 / 55.7 / 52.6 on SET1 splits I-IV, versus best prior baseline IIC at 43.8 / 40.1 / 44.5 / 45.6.
- Averaged across splits, clustering accuracy improves by about 10% over state-of-the-art methods, per the abstract.
- Under task-agnostic protocol on SET1, labeled classification accuracy improves by about 8% on average across four splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add paired text branch with intra-modal consistency weighting of pseudo-labels | NCD on MIMIC-CXR-JPG (image branch) · SET1 and SET2, task-aware protocol | cluster accuracy on unlabeled test: about 10% average gain over SOTA per abstract; SET1 I 56.9 vs 43.8 (IIC) | Helped |
| intra-modal consistency weighting instead of pseudo-label averaging (ablation) | two-branch pseudo-label generation · SET1 four splits, task-aware | cluster accuracy on unlabeled test: e.g. split I 56.9 vs 54.8 (PL avg.) | Helped |
| intra-modal consistency weighting instead of logits averaging (ablation) | two-branch pseudo-label generation · SET1 four splits, task-aware | cluster accuracy on unlabeled test: e.g. split I 56.9 vs 52.7 (Logits avg.) | Helped |
| multi-modal knowledge distillation from frozen text branch to image branch (UNO MMKD) | UNO · SET1 and SET2, task-aware | cluster accuracy: e.g. SET1 I 42.4 vs 43.6 for UNO | No clear effect |
| add BYOP novel class prior estimation to the proposed method | Ours · SET1 task-aware, split II | cluster accuracy: 58.8 vs 53.7 | Helped |
| text-augmented NCD with intra-modal weighting | task-agnostic protocol on SET1 · SET1 task-agnostic, labeled test classification | classification accuracy: about 8% average improvement across four splits | Helped |
| text-augmented NCD with intra-modal weighting | task-agnostic protocol on SET1 · SET1 task-agnostic, unlabeled test clustering | cluster accuracy: about 5% gain on splits II and IV; close to best on I and III | Mixed |
Surprising findings
- Simple averaging of pseudo-labels or logits from both modalities underperforms consistency-weighted fusion, and the text-to-image multi-modal knowledge distillation (UNO MMKD) gives no clear gain.
Limitations
- Text is only used during training; test-time performance relies on the image branch alone.
- Evaluation limited to 11 selected MIMIC-CXR-JPG classes and eight splits from one dataset.
Experimental setup
Datasets: MIMIC-CXR-JPG Chest ImaGenome
Benchmarks: MIMIC-CXR-JPG SET1 splits I-IV MIMIC-CXR-JPG SET2 splits I-IV
Metrics: clustering accuracy classification accuracy
Baselines: K-means AutoNovel NCL UNO IIC ComEx UNO MMKD BYOP
Builds on: UNO BYOP GLoRIA IIC Sinkhorn-Knopp
Closest papers by meaning
- Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.87)
- CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification (CVPR 2024 · similarity 0.85)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.85)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.85)
- GRAPE: Graph-Augmented Prototype Explanations for Interactive Medical Image Diagnosis (NeurIPS 2026 · similarity 0.84)
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.