Research map / Learning theory and algorithms / Fairness
Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare
Max Lamparth, Declan Grabb, Amy Franks, Scott Gershan, Kaitlyn N Kunstman, Aaron Lulla, Monika Drummond Roots, Manu Sharma, Aryan Shrivastava, Nina Vasan, Colleen Waickman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| mental-health or clinical fine-tuning (MMedS-Llama-3-8B) vs base Llama 3.1 8B (ablation) | MMedS-Llama-3-8B · MENTAT | accuracy: 0.65 -> 0.58 | No clear effect |
| fine-tuned open-source health models vs Llama2/Llama3 counterparts (ablation) | health fine-tuned LMs · MENTAT | accuracy | No clear effect |
| use larger closed-source models vs smaller open-source models | closed-source LMs · MENTAT | accuracy: Llama 3.1 8B 0.65 -> o1 0.81 | Improved |
| ambiguous categories (triage, documentation) vs unambiguous categories | all LMs · MENTAT | accuracy: o1 diagnosis 0.96 -> triage 0.46 | Hurt |
| vary patient gender: male vs female coded | LMs · MENTAT monitoring/triage/documentation | accuracy: +10% monitoring, +8% triage, +10% documentation for male-coded | Mixed |
| describe patient as African American vs White | LMs · MENTAT diagnosis | accuracy: +5% | Mixed |
| describe patient as Native American vs other ethnicities | LMs · MENTAT treatment | accuracy: +7 to 11% | Mixed |
| use hierarchical Bradley-Terry model instead of regular for preference labels (ablation) | annotation processing · MENTAT triage/documentation | creator answer in top-3: top-5 -> top-3 | Improved |
Closest papers by meaning
- Position: Beyond Assistance – Reimagining LLMs as Ethical and Adaptive Co-Creators in Mental Health Care (ICML 2025 · similarity 0.86)
- TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots (NeurIPS 2026 · similarity 0.84)
- CLR-voyance : Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics (NeurIPS 2026 · similarity 0.84)
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias (NeurIPS 2024 · similarity 0.84)
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.84)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
