Research map / Architectures and efficiency / Attention and transformers
HDformer: A Higher-Dimensional Transformer for Detecting Diabetes Utilizing Long-Range Vascular Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HDformer, a Swin-based Transformer with Time Square Attention (1D PPG reshaped into 2D tokens) and a gated mixture-of-experts over dynamic patch sizes, to detect diabetes from 10-minute PPG on MIMIC-III.
Problem
Non-invasive diabetes detection from PPG has limited accuracy, and standard Transformers cannot efficiently process long-range (10+ minute) vascular waveforms.
Approach
Time Square Attention groups adjacent 1D PPG samples into 2D tokens with dynamic patch sizes, feeding them to a 2D Transformer (Swin) whose outputs are combined by a gated mixture-of-experts, claiming linear complexity in token count.
Key results
- HDformer on MIMIC-III 10-minute PPG: sensitivity 98.4, accuracy 97.3, specificity 92.8, AUC 0.929, versus best prior baseline AUC 0.830 (Srinivasan and Foroozan 2021).
- Swin-based TSA outperforms ViT-based TSA (AUC 0.929 vs 0.910) and 1D Informer at 30s achieves AUC 0.806.
- Multimodal PPG+reconstructed ECG reduces required input to 5 minutes with accuracy 95.1 and AUC 0.917.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase 1D input length from 8s to 30s (ablation) | Informer · MIMIC-III | accuracy: 72.5 -> 74.5 | Helped |
| increase 1D input length from 30s to 60s (ablation) | Informer · MIMIC-III | accuracy: 74.5 -> 68.8 | Hurt |
| increase TSA input length from 30s to 10m (ablation) | HDformer · MIMIC-III | accuracy: 75.1 -> 97.3 | Helped |
| replace 1D attention with TSA (ablation) | Informer · MIMIC-III | accuracy: 72.5 -> 85.6 | Helped |
| use patch size 4T instead of T/4 (ablation) | HDformer TSA · MIMIC-III | accuracy: 77.5 -> 89.1 | Helped |
| combine dynamic patch sizes with gated MoE (ablation) | HDformer TSA · MIMIC-III | accuracy: 89.1 -> 97.3 | Helped |
| use 2D image representations instead of numerical tensors (ablation) | HDformer TSA · MIMIC-III | accuracy: 75.1 -> 62.8 | Hurt |
| replace Swin Transformer with ViT (ablation) | TSA · MIMIC-III | accuracy: 97.3 -> 95.9 | Hurt |
| add 256hz sampling to 5m PPG (ablation) | HDformer · MIMIC-III diabetes detection | accuracy: 82.9 -> 86.1 | Helped |
| add reconstructed ECG multimodal input to 5m, 256hz PPG (ablation) | HDformer · MIMIC-III diabetes detection | accuracy: 86.1 -> 95.1 | Helped |
Surprising findings
- Image-based 2D representations performed worse than 1D-to-numerical 2D tensors, and longer image inputs degraded performance further.
- On 1D Informer, extending input from 30s to 60s reduced performance, which the authors attribute to computational overload.
Limitations
- Evaluated only on MIMIC-III (single-center ICU data) and one additional dataset; no prospective or self-collected validation reported.
- Linear complexity claim is argued theoretically, not benchmarked empirically.
- Long-range 10-minute PPG input still requires substantial signal length for practical wearable use.
Experimental setup
Datasets: MIMIC-III Liang et al. 2018 diabetes dataset
Benchmarks: MIMIC-III
Metrics: sensitivity accuracy specificity AUC
Baselines: Moreno et al. 2016 Reddy et al. 2017 Hettiarachchi and Chitraranjan 2019 Qawqzeh et al. 2020 Avram et al. 2020 Wang et al. 2020a Srinivasan and Foroozan 2021 Informer ViT
Builds on: Swin Transformer Vision Transformer Informer mixture-of-experts
Closest papers by meaning
- FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing (NeurIPS 2024 · similarity 0.83)
- Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification (NeurIPS 2024 · similarity 0.83)
- Constraint Latent Space Matters: An Anti-anomalous Waveform Transformation Solution from Photoplethysmography to Arterial Blood Pressure (AAAI 2024 · similarity 0.82)
- DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation (AAAI 2024 · similarity 0.82)
- PaPaGei: Open Foundation Models for Optical Physiological Signals (ICLR 2025 · similarity 0.81)
- Boltzmann Attention Sampling for Image Analysis with Small Objects (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.