Hierarchical transformer for long multimodal health records, pretrained with BYOL on 2.8M UK patients to predict 5-year disease risk.
No providers recorded yet. Browse all providers
A UK primary-care record accumulates for decades: every prescription renewal, blood-pressure reading, lab test and smoking-status update, as well as diagnoses. In this study a third of patients had more than 256 records before prediction time. Self-attention cost grows with the square of sequence length, so earlier EHR transformers used only diagnoses or truncated the history to the most recent few hundred events, hiding the slow build-up of risk factors that precedes heart failure or kidney disease.
Hi-BEHRT (hierarchical BEHRT), developed by the Deep Medicine group at the Oxford Martin School, University of Oxford, removes that limit by dividing the problem in two. A sliding window cuts the full record into short overlapping segments. A local transformer summarises each segment, since events close in time tend to be related. A second transformer then attends across the segment summaries to capture dependencies that span years. Posted to arXiv in June 2021 and published in the IEEE Journal of Biomedical and Health Informatics in 2023, it is the same group's successor to BEHRT.
Masked-token pretraining, the usual choice for EHR transformers, is awkward for a hierarchical model because the upper transformer never sees raw tokens, only latent segment summaries. So the authors adapted BYOL (Bootstrap Your Own Latent), a contrastive method from computer vision, to pretrain the whole architecture end to end. The pretrained encoder is then fine-tuned for individual disease-risk tasks.
Hi-BEHRT was trained on the Clinical Practice Research Datalink (CPRD), de-identified UK primary-care records covering about 7% of the population, linked to Hospital Episode Statistics. The cohort was patients aged 16 and older contributing data between 1985 and September 2015. The 4,063,811 eligible patients were split 60/10/30 into training (2,438,352), tuning (406,381) and validation (1,219,078) cohorts; the training and tuning cohorts together — 2,844,733 patients, about 2.8 million — were the ones used for BYOL pretraining, leaving the validation cohort unseen by it. The vocabulary covered 1,995 diagnosis, 378 medication, 275 test and 960 procedure codes plus binned vital-sign categories. Each transformer stack has 4 layers, hidden size 150, 6 attention heads and intermediate size 108. BYOL pretraining used a target-network decay of 0.996 and ran for 35 epochs over 10 days on two P100 GPUs, short of the planned 1,000.
Evaluation was 5-year incident risk of heart failure, diabetes, chronic kidney disease and stroke. Hi-BEHRT reached AUROC of 0.96, 0.93, 0.93 and 0.90 and AUPRC of 0.77, 0.74, 0.79 and 0.79, ahead of BEHRT, Med-BERT and CEHR-BERT retrained on the same data. The margin was largest for patients whose records exceeded 256 events. With 5% of the training data, pretraining raised stroke AUPRC from 0.24 to 0.41 and CKD AUPRC from 0.26 to 0.33. On heart failure, pretraining helped most with 1–10% of the training data, and performance with or without it plateaued at around half the training set.
Hi-BEHRT suits epidemiologists and clinical-informatics teams building risk models from routine records, particularly for chronic conditions such as cardiovascular, metabolic and renal disease, where risk builds over years and long histories are the norm. The pretraining results matter most for rare outcomes and small labelled cohorts. The public code covers CPRD preprocessing, the hierarchical model, BYOL pretraining and fine-tuning, but no pretrained checkpoint is distributed, so users need their own large EHR corpus to pretrain before fine-tuning.
Hi-BEHRT showed that sequence length, not only vocabulary, limits EHR transformers, and that a hierarchical design recovers information from decades-long records at lower cost than a flat model. It also showed early that contrastive pretraining transfers to sequential clinical data. The evaluation has clear limits: all results come from held-out patients in one UK source rather than external cohorts, pretraining stopped at 35 of 1,000 planned epochs, and the lack of a released checkpoint or code license leaves reuse dependent on retraining under a CPRD data licence.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.