Research map / Large language models / Language, safety and interpretability
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Cross-Care, a benchmark that compares LLM logit-based disease-demographic associations with The Pile co-occurrence counts and real US prevalence data across four languages.
Problem
LLMs may encode demographic biases from pre-training data about disease prevalence, and it is unclear whether their representations are grounded in real-world epidemiological prevalence across demographic subgroups and languages.
Approach
Measure model logits for ten templated statements linking diseases and demographic subgroups, then compare the resulting rankings via Kendall's tau against co-occurrence counts in The Pile and CDC NHIS real-world prevalence, across models, alignment stages, and languages.
Key results
- No model's disease logit rankings reached tau > 0.35 with real-world prevalence rankings for gender or race/ethnicity (median tau -0.05), indicating no real-world grounding.
- Pythia and Mamba logit rankings matched The Pile co-occurrence top-ranked subgroup more often than real prevalence, and larger models followed pre-training distributions less closely.
- Alignment methods (SFT, DPO, RLHF) and biomedical continued pre-training did not correct the base Llama2 70b toward real-world prevalence and sometimes increased bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use model logits to rank demographic disease prevalence | Models in the wild · Real-world disease prevalence, gender and race/ethnicity | Kendall's tau: Min = -0.73, Max = 0.33, Median = -0.05, Avg = -0.06, Var = 0.03 | Hurt |
| Increase Pythia or Mamba model size | Pythia and Mamba · The Pile subgroup rankings across 89 diseases | Agreement of top demographic ranks with co-occurrence | Hurt |
| Compare Pythia and Mamba logits to The Pile co-occurrence rankings | Pythia and Mamba · Real-world disease prevalence, gender and race/ethnicity | Kendall's tau | Mixed |
| Increase disease co-occurrence frequency quartile | Pythia and Mamba · The Pile disease co-occurrence quartiles | Logit correlation with co-occurrence | No clear effect |
| Change the co-occurrence token window among 50, 100, and 250 tokens (ablation) | Cross-Care co-occurrence rankings · Disease-demographic pairs | Top disease rank: 50, 100, and 250 | No clear effect |
| Apply alignment methods or in-domain continued pre-training | Llama2 70b series · Real-world disease prevalence | Accuracy of demographic prevalence representation | No clear effect |
| Apply alignment methods to Llama2 70b | Llama2 70b · English gender and race/ethnicity rankings | Subgroup preference | Mixed |
| Apply SFT or DPO alignment | Llama2 70b · Race/ethnicity subgroup rankings | Kendall's tau rank-order stability: δ>=0.8 | No clear effect |
| Use RLHF chat tuning or medical continued pre-training | Llama2 70b · Race/ethnicity subgroup rankings | Rank-order stability | Mixed |
| Use PPO+DPO chat alignment instead of the base model | Qwen1.5-7b · English race/ethnicity subgroup preference | Top subgroup preference | Mixed |
| Evaluate Mistral and Llama models in Chinese rather than French | Mistral and Llama series · Gender subgroup preference | Top subgroup preference | Mixed |
| Evaluate Qwen1.5 in different languages | Qwen1.5 · Gender subgroup preference | Top subgroup preference | Mixed |
Surprising findings
- Larger models followed the pre-training co-occurrence distribution less closely than smaller ones in the controlled Pythia and Mamba experiments.
- Biomedical continued pre-training (Meditron, Bio-mistral) and alignment did not improve agreement with real-world prevalence.
Limitations
- No NER tagging, so keyword co-occurrences may include non-referential uses (e.g., 'white blood cells').
- Only 15 of 89 diseases had official CDC prevalence data, and only male/female gender subgroups.
- Race/ethnicity and gender categories are simplistic, and results depend on template design.
Experimental setup
Datasets: The Pile RedPajama CDC National Health Interview Survey
Benchmarks: Cross-Care
Metrics: Kendall's tau average logit top-rank agreement
Baselines: Infini-gram
Builds on: The Pile Pythia Mamba HolisticBias Infini-gram
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.87)
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.87)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.86)
- RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios? (AAAI 2025 · similarity 0.86)
- Multilingual Medical Language Models: A Path to Improving Lay Health Worker Effectiveness (Student Abstract) (AAAI 2024 · similarity 0.85)
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.