Research map / Architectures and efficiency / Attention and transformers
How Should Transformers Encode Numeric Values in Electronic Health Records?
Maria Elkjær Montgomery, Christian Igel, Mikkel Fruelund Odgaard, Martin Sillesen, Mads Nielsen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FiLM value-concept conditioning instead of binned token encodings | EHR transformer · synthetic addition, multiplication, polynomial tasks | information efficiency | Improved |
| use combination (continuous token insertion) vs FiLM | EHR transformer · precision-sensitive arithmetic tasks | information efficiency | Hurt |
| add binning before linear projection (combined binning) | continuous token encoding · low-data synthetic classification | AUC | Improved |
| use discretised tokens instead of FiLM | EHR transformer · sharp-edge detection | information efficiency | Improved |
| add laboratory values to lab-name-only baseline | EHR transformer · breast cancer, lung cancer, stroke prediction | AUROC | Mixed |
| use concatenation encoding for lab values | EHR transformer · lung cancer prediction | AUROC | Hurt |
| truncate FiLM context to match two-token binned methods (ablation) | FiLM · clinical prediction tasks | AUROC: p > 0.05 | No clear effect |
| vary MSE loss weight w in 0.001 to 1.5 (ablation) | combination method · synthetic classification n=100,000 | information efficiency | No clear effect |
| fit power law instead of log or linear for optimal bins vs N | bin-count heuristic · synthetic classification | R^2: 0.74 -> 0.83 | Improved |
Closest papers by meaning
- RNED: Rotary Number Encoding and Decoding for Medical VLMs (CVPR 2026 · similarity 0.84)
- Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (ICLR 2025 · similarity 0.82)
- One Loss to Rule Them All: Marked Time-to-Event for Structured EHR Foundation Models (NeurIPS 2026 · similarity 0.82)
- Efficient numeracy in language models through single-token number embeddings (ICML 2026 · similarity 0.82)
- Transformers Can Do Arithmetic with the Right Embeddings (NeurIPS 2024 · similarity 0.82)
- ConSequence: Synthesizing Logically Constrained Sequences for Electronic Health Record Generation (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
