Research map / Vision-language and multimodal / Human and embodied modeling
Multimodal Medical Code Tokenizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MedTok, a multimodal vector-quantized tokenizer for medical codes that fuses text descriptions and PrimeKG subgraph embeddings into a unified codebook, and plugs it into five EHR models and a medical QA pipeline.
Problem
Standard EHR tokenizers treat medical codes (over 600,000 across eight coding systems) as isolated text tokens, losing their textual definitions and relational/ontological context, which hurts downstream EHR prediction.
Approach
Encode each code's text with a frozen language model and its knowledge-graph subgraph with a graph encoder, then quantize both into a unified codebook split into text-specific, graph-specific, and shared regions, with KL and InfoNCE-style packing losses to align cross-modal information.
Key results
- Swapping in MedTok improves AUPRC across all five EHR models: +4.10% on MIMIC-III, +4.78% on MIMIC-IV (drug recommendation).
- On EHRShot out-patient tasks, MedTok gives an average +11.32% AUPRC gain on mortality prediction within operational outcomes.
- Adding both shared and specific losses improves average AUPRC by 3.9%, 5.7%, and 9.1% over the VQ-only loss on the three datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard medical-code tokenization with MedTok | TransformEHR · MIMIC-III mortality prediction | AUPRC: 0.207 (0.012) -> 0.246 (0.044) | Helped |
| Replace standard medical-code tokenization with MedTok | TransformEHR · MIMIC-IV mortality prediction | AUPRC: 0.042 (0.012) -> 0.058 (0.007) | Helped |
| Replace standard medical-code tokenization with MedTok | TransformEHR · MIMIC-III drug recommendation | AUPRC: 0.533 (0.030) -> 0.580 (0.035) | Helped |
| Replace standard medical-code tokenization with MedTok | TransformEHR · MIMIC-IV drug recommendation | AUPRC: 0.612 (0.046) -> 0.661 (0.092) | Helped |
| Replace standard medical-code tokenization with MedTok | TransformEHR · EHRShot mortality prediction | AUPRC: 0.344 (0.030) -> 0.379 (0.034) | Helped |
| Replace standard medical-code tokenization with MedTok | TransformEHR · EHRShot hyperlipidemia diagnosis assignment | AUPRC: 0.305 (0.021) -> 0.333 (0.018) | Helped |
| Add cross-modal packing and KL losses to VQ-only training (ablation) | MedTok · MIMIC-III, average across tasks | AUPRC: 0.373 -> 0.379 | Helped |
| Add cross-modal packing and KL losses to VQ-only training (ablation) | MedTok · MIMIC-IV, average across tasks | AUPRC: 0.387 -> 0.409 | Helped |
| Add cross-modal packing and KL losses to VQ-only training (ablation) | MedTok · EHRShot, average across tasks | AUPRC: 0.287 -> 0.314 | Helped |
| Add modality-specific packing to VQ-only training (ablation) | MedTok · MIMIC-III, average across tasks | AUPRC: 0.373 -> 0.382 | Helped |
| Add modality-specific packing to VQ-only training (ablation) | MedTok · MIMIC-IV, average across tasks | AUPRC: 0.387 -> 0.402 | Helped |
| Add modality-specific packing to VQ-only training (ablation) | MedTok · EHRShot, average across tasks | AUPRC: 0.287 -> 0.366 | Helped |
Limitations
- Improvements on some tasks have overlapping standard deviations with baselines (e.g., readmission on MIMIC-III).
- Text definitions were refined with GPT-4 turbo, introducing LLM-generated content into the code descriptions.
- Medical QA evaluation is limited to three datasets and three LLMs with prefix tuning on MedMCQA.
Experimental setup
Datasets: MIMIC-III MIMIC-IV EHRShot PrimeKG MedMCQA UMLS DrugBank
Benchmarks: MIMIC-III MIMIC-IV EHRShot MMLU MedDDx AfrimedQA
Metrics: AUPRC accuracy AUC-ROC
Baselines: ETHOS GT-BEHRT MulT-EHR TransformEHR BEHRT BERT tokenizer VQGraph
Builds on: VQGraph BEHRT TransformEHR ETHOS GT-BEHRT MulT-EHR PrimeKG
Closest papers by meaning
- Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (ICLR 2025 · similarity 0.85)
- MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records (AAAI 2024 · similarity 0.85)
- SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark (NeurIPS 2024 · similarity 0.84)
- BoxLM: Unifying Structures and Semantics of Medical Concepts for Diagnosis Prediction in Healthcare (ICML 2025 · similarity 0.84)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.84)
- MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.