Research map / Large language models / AI and society
Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a guided deferral system where an instruction-tuned open-source LLM parses clinical reports, combines verbalised and hidden-state predictions, defers uncertain cases to humans with LLM-generated guidance, and introduces ECE_Imb for calibration under class imbalance.
Problem
Deploying LLMs for clinical report classification in healthcare requires trustworthy, privacy-preserving, computationally modest systems that can defer uncertain cases to humans with useful guidance, while standard calibration metrics (ECE, ACE) are unreliable under class imbalance.
Approach
Combine verbalised probabilities from the LLM's generated text with a hidden-state classifier on the final-layer embeddings, average them into a combined prediction for deferral, use guard-railed instruction-tuning of a small open-source LLM (QLoRA) on outputs of a larger model, and provide dialectic reasoning as guidance to human deferrals; also propose ECE_Imb, a re-weighted ECE mixing bin propo
Key results
- On OSCLMRIC, the combined prediction of the instruction-tuned 13B model (mu-hat IT-13B) achieved the highest F1 of 0.94 ± 0.01, with a statistically significant improvement over the verbalised and hidden-state sources alone (p<0.01).
- On MIMIC-500, the combined prediction of the 13B base model reached a micro-averaged F-score of 93.17 and macro-averaged 90.52, exceeding GPT-3.5 (92.68 / 89.64) and falling just short of GPT-4 (93.87 / 92.79).
- In a pilot study with 20 participants deferring 30 uncertain cases, accuracy improved for all participants with guidance (paired one-sided Wilcoxon signed-rank test, p<0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune the 13B model | Verbalised prediction · OSCLMRIC test | F1-score: 0.83 ± 0.01 -> 0.88 ± 0.02 | Helped |
| Instruction-tune the 13B model | Hidden-state prediction · OSCLMRIC test | AUARC: 0.997191 ± 0.0002 -> 0.997409 ± 0.0002 | Helped |
| Combine verbalised and hidden-state predictions (ablation) | IT-13B · OSCLMRIC test | F1-score: 0.88 ± 0.02; 0.93 ± 0.01 -> 0.94 ± 0.01 | Helped |
| Use hidden-state predictions instead of verbalised predictions (ablation) | BASE-13B · OSCLMRIC test | AUARC: 0.897460 ± 0.0063 -> 0.994715 ± 0.0006 | Helped |
| Use combined predictions instead of verbalised predictions (ablation) | BASE-13B · OSCLMRIC test | F1-score: 0.61 ± 0.01 -> 0.82 ± 0.03 | Helped |
| Classify with combined predictions while deferring with hidden-state predictions | IT-13B · OSCLMRIC test, deferral rates below 5% | AUARC | Helped |
| Classify with combined predictions while deferring with hidden-state predictions | IT-13B · OSCLMRIC test, deferral rates at least 5% | AUARC | No clear effect |
| Provide LLM guidance on deferred cases | Human decision-makers · OSCLMRIC pilot study | accuracy: p<0.01 | Helped |
| Combine verbalised and hidden-state predictions | BASE-13B · MIMIC-500 impressions | Micro-Avg. F-score: 92.68 -> 93.17 | Helped |
Surprising findings
- The hidden-state classifier on the instruction-tuned 13B model had higher deferral performance (AUARC 0.9974) than the 70B base-model verbalised output, despite being a smaller model.
- Standard ECE and ACE are misleading under class imbalance, with ECE concentrating most bin weight on the first bin.
Limitations
- OSCLMRIC is a single-site, small dataset (100 reports, 1,800 examples) and cannot be shared, limiting external validation.
- The pilot study used 20 researchers without specialist spinal expertise, not clinicians, and only 30 deferred cases.
- Comparisons against GPT-3.5 and GPT-4 are on MIMIC-500 only, and the proprietary models were not run on the private OSCLMRIC data.
Experimental setup
Datasets: OSCLMRIC MIMIC-500 MIMIC-CXR-JPG CheXpert labels
Benchmarks: OSCLMRIC test split MIMIC-500
Metrics: precision recall F1-score ECE ACE ECE_Imb AUARC relative seconds per generation GPU memory error rate
Baselines: GPT-3.5 GPT-4 Tulu V2 70B (base) unguided human decision-making
Builds on: Tulu V2 DPO QLoRA Expected Calibration Error Adaptive Calibration Error Learning-to-Defer
Closest papers by meaning
- CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models (NeurIPS 2024 · similarity 0.88)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.84)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.84)
- CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs (NeurIPS 2025 · similarity 0.84)
- MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare (NeurIPS 2026 · similarity 0.84)
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.