Research map / Large language models / Retrieval and knowledge
A teacher-teacher framework for clinical language representation learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a teacher-teacher framework in which a lightweight LINE module aligns two frozen pretrained clinical/general LLM embedding spaces (BGE or GPT-4 with CODER) using paired clinical notes and UMLS-linked concept lists, enabling concept-only proxy representations of notes.
Problem
Clinical notes are privacy-restricted and hard to share, while domain-specific and general LLMs hold complementary knowledge; continual pretraining of clinical models is costly. The paper asks how to let pretrained models exchange knowledge and how to use shareable concept lists as proxies for clini
Approach
Instead of further training teacher LLMs, train only a small projection module (a fully-connected layer for the general teacher and a multihead graph attention layer over UMLS relation graphs for the domain teacher) with a triplet alignment loss weighted by text coverage plus a relational contrastive loss, using a two-stage residual-recovery schedule to add missed concepts.
Key results
- On a holdout set of 275K text-concept pairs, LINE (BGE+CODER) reduced mean rank to 1.437 from 5.526 for BGE and achieved Top10@Acc of 0.997.
- On i2b2 2014 test F1, LINE reached 98.1 versus 97.1 for BGE and 94.9 for UmlsBERT.
- For renal recurrence classification, LINE-projected concept-only embeddings narrowed the BGE sentence-vs-concept F1 gap from 0.062 to 0.037.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align BGE and CODER representations with LINE | BGE+CODER · Holdout radiology reports | mean rank: 5.526 -> 1.437 | Helped |
| align BGE and CODER representations with LINE | BGE+CODER · Holdout radiology reports | Top10@Acc: 0.845 -> 0.997 | Helped |
| add LINE alignment to BGE embeddings | BGE+CODER · UMLS relation classes | AUC | Mixed |
| apply LINE projection to token-level BGE embeddings | i2b2 NER · i2b2 2006 test | F1: 97.2 -> 98.1 | Helped |
| apply LINE projection to token-level BGE embeddings | i2b2 NER · i2b2 2014 test | F1: 97.1 -> 98.1 | Helped |
| add LINE projection for concept-list embeddings | BGE+CODER · Renal cancer recurrence, concepts | F1: 0.711 -> 0.734 | Helped |
| add LINE projection for concept-list embeddings | GPT-4+CODER · Renal cancer recurrence, concepts | F1: 0.694 -> 0.714 | Helped |
| project CODER embeddings into BGE space instead of using LINE (ablation) | CODER+BGE · Holdout radiology reports | mean rank: 1.437 -> 2.509 | Hurt |
| project BGE embeddings into CODER space instead of using LINE (ablation) | BGE+CODER · Holdout radiology reports | mean rank: 1.437 -> 3.281 | Hurt |
| project CODER embeddings into BGE space instead of using LINE (ablation) | CODER+BGE · Renal cancer recurrence, concepts | F1: 0.734 -> 0.666 | Hurt |
| project BGE embeddings into CODER space instead of using LINE (ablation) | BGE+CODER · Renal cancer recurrence, sentences | F1: 0.771 -> 0.769 | No clear effect |
Surprising findings
- The general-purpose teachers (BGE, GPT-4) outperform CODER on most tasks, yet alignment with CODER still improved their performance on some relation classes and tasks.
Limitations
- Relies on paired data, which may be scarce in some settings.
- Requires a comprehensive knowledge graph (UMLS); performance may drop where it is incomplete.
- Code and the non-public renal cancer dataset are not released.
Experimental setup
Datasets: MIMIC-IV UMLS 2022AB i2b2 2006 i2b2 2014 UMLS 2020AA relation triplets
Benchmarks: i2b2 2006 i2b2 2014 UMLS concept relation pairs (may treat/prevent, classifies, DDX, method of, causative) renal cancer recurrence detection (in-house, 288 sentences)
Metrics: mean rank mean reverse rank Top10@Acc AUC F1 precision recall accuracy
Baselines: BioBERT Clinical BioBERT SapBERT PubMedBERT UmlsBERT GPT-4 embeddings BGE CODER
Builds on: CODER BGE CLIP SapBERT
Closest papers by meaning
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.85)
- ClinTutor-R1: Advancing Scalable and Robust One-to-Many Alignment in Clinical Socratic Education (ICML 2026 · similarity 0.83)
- Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling (ICML 2025 · similarity 0.82)
- Large Language Models Are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales (AAAI 2024 · similarity 0.82)
- DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models (NeurIPS 2024 · similarity 0.82)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.