BERT-style transformer for longitudinal health records, pretrained on 1.6M UK patients' diagnosis histories to predict onset of 301 conditions.
No providers recorded yet. Browse all providers
A general practitioner reading a patient's chart does not see a flat list of diagnoses. They see that asthma came at age six, hypertension at fifty, and a cluster of new codes in the last two visits, and that ordering and timing is most of what makes the chart predictive. Earlier deep models of electronic health records (EHR) captured parts of this with recurrent networks or convolutions over code sequences, but struggled with long gaps between related events and with the irregular spacing of real visits.
BEHRT (BERT for EHR), developed by the Deep Medicine group at the Oxford Martin School, University of Oxford, and first posted in July 2019, carries the transformer recipe from natural language over to structured records. Each diagnosis code is a word, each visit a sentence, and a patient's whole history a document. Self-attention lets any diagnosis attend to any other regardless of how many visits separate them, and a masked-language-model objective pretrains the network on unlabeled histories before it is fine-tuned to forecast future diagnoses. The work was published in Scientific Reports in 2020.
The model is an early template for the pretrain-then-fine-tune approach to structured clinical data that later entries such as CEHR-BERT extend with richer concept vocabularies and explicit time tokens.
CLS
start token and SEP separators, and fed to a bidirectional transformer encoder.BEHRT was trained on the Clinical Practice Research Datalink (CPRD), UK primary-care records from 674 GP practices linked to Hospital Episode Statistics. Starting from about 8 million patients, the authors kept those eligible for linkage, meeting CPRD quality flags and with at least five visits, leaving 1.6 million. Read codes (primary care) and level-4 ICD-10 codes (hospital) were mapped to 301 CALIBER disease codes. Only diagnoses and age were used as inputs, although the architecture admits further concepts through additional embeddings. Bayesian optimisation selected an encoder of 6 layers, 12 attention heads, hidden size 288 and intermediate size 512, pretrained for 100 epochs to a masked-token precision of 0.66. Patients were split 80/20 into training and test sets.
The pretrained encoder was then fine-tuned separately for three tasks: diagnoses at the next visit (699,000 patients), within 6 months (391,000) and within 12 months (342,000). Average precision scores were 0.462, 0.525 and 0.506 respectively, against 0.360–0.393 for Deepr and 0.382–0.417 for RETAIN, with AUROC around 0.95. On first-incidence diagnoses, those not already in a patient's history, BEHRT reached an APS of 0.216–0.228 against at most 0.115 for the baselines. The 288-dimensional disease embeddings cluster by clinical chapter and separate sex-specific conditions without sex being an input. A clinical researcher judged 76% of the top-10 embedding neighbourhoods for 87 common diseases clinically valid.
BEHRT serves epidemiologists and clinical informaticians who want multi-disease risk forecasts from routine records, and anyone who needs dense patient or disease vectors for downstream analysis such as patient similarity, phenotype clustering or trajectory mapping in multimorbidity research. The public repository provides the data formatting, masked-language-model pretraining and next-visit fine-tuning code, but no pretrained weights: CPRD data are available only under licence with protocol approval, so users must pretrain on their own EHR data before applying the model.
BEHRT was among the first demonstrations that BERT-style pretraining transfers to structured diagnosis sequences at national scale, and its framing of patient histories as documents became a common starting point for EHR foundation models. The Oxford group extended it in Hi-BEHRT, a hierarchical variant for long multimodal records, and independent groups have built on the name and design, including Novartis's ExBEHRT. Its limitations are those of the original study: a diagnosis-only vocabulary, a single UK data source, evaluation on held-out patients from that same source rather than external cohorts, and no released checkpoint, which leaves reuse dependent on retraining.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.