Research map / Large language models / Language, safety and interpretability
RaLEs: a Benchmark for Radiology Language Evaluations
Juan Manuel Zambrano Chaves, Nandita Bhaskhar, Maayane Attias, Jean-Benoit Delbrouck, Daniel Rubin, Andreas Markus Loening, Curtis Langlotz, Akshay S Chaudhari
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 10% of training labels instead of full (ablation) | BERT-base · RaLEs NLU | average score: 81.8 -> 64.2 | Hurt |
| domain-specific clinical/radiology pretraining vs general English | encoder models · RaLEs NLU | average score | Improved |
| replaced-token-detection objective (ELECTRA) vs masked LM only | ELECTRA-base · RaLEs NLU | average score: 82.0 | Improved |
| add document-link prediction objective (BioLinkBERT vs PubMedBERT) | PubMedBERT-base · RaLEs NLU | average score: 81.9 -> 83.3 | Improved |
| increase parameters from ~100M to ~300M | BERT/RoBERTa/DeBERTa/BioLinkBERT · RaLEs NLU | average score: 83.0 -> 82.7 | No clear effect |
| larger BERT (110M to 340M) | BERT · RaLEs NLU | average score: 81.8 -> 81.4 | No clear effect |
| gains on GLUE or BLURB as predictor of radiology NLU | general and biomedical models · RaLEs NLU | average score: 10 percentage point improvements in GLUE or BLURB lead to a 1-2 percent improvement | No clear effect |
| radiology domain adaptation (RadBERT) on other benchmarks | RadBERT1,2 · BLURB | score | Hurt |
| fine-tune with frozen LM weights (ablation) | BERT-base · RaLEs NLU | average score: 81.8 -> 69.4 | Hurt |
| clinical GatorTron vs general ELECTRA-base for summarization | ELECTRA-base · MEDIQA 2021 | ROUGE-2: .238 -> .250 | Improved |
| clinical GatorTron vs ELECTRA-base for factuality | ELECTRA-base · BioNLP 2023 | RadGraph F1: .229 -> .283 | Improved |
| out-of-distribution test (Stanford) instead of in-distribution (Indiana) | GatorTron · summarization | ROUGE-2: .262 -> .238 | Hurt |
Closest papers by meaning
- LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.89)
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.89)
- HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation (AAAI 2025 · similarity 0.88)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.86)
- RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis (NeurIPS 2025 · similarity 0.85)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
