BERT for structured EHR patient journeys that stacks procedures, labs, BMI and smoking status under each diagnosis so sequence length stays fixed.
No providers recorded yet. Browse all providers
A patient admitted for a lung infection may leave the hospital with one diagnosis code, four procedures, six laboratory panels, a recorded BMI and a smoking status. Transformer models for structured health records usually treat that history as a sentence: every concept becomes a token in one long horizontal sequence. Adding procedures or labs then multiplies sequence length, and with it memory and training cost. BEHRT, the first BERT adaptation to electronic health records, sidestepped the problem by modelling only diagnoses and age, discarding most of what the chart records.
ExBEHRT keeps the diagnosis sequence as the backbone and stacks everything else underneath it. Each visit gets as many horizontal slots as it has diagnoses; procedures and lab types are wrapped into those slots row by row and padded to a fixed height, while visit-level values such as BMI, smoking status and gender are copied across the visit's slots. Every concept type has its own learned embedding, and the rows are summed element-wise into a single vector per slot before entering the encoder. The input length is therefore set by the number of diagnoses alone, however many other concept types are added.
The model was developed by Maurice Rupp, Oriane Peter and Thirupathi Pattipaka at Novartis Oncology AG in Basel, part of Novartis. It was presented at the ICLR 2023 Workshop on Trustworthy Machine Learning for Healthcare and published in the workshop proceedings in Springer's Lecture Notes in Computer Science.
The encoder follows BEHRT's configuration: 6 transformer layers, 12 attention heads and a 288-dimensional embedding. Pretraining uses masked language modelling on diagnosis codes, masking 15% of them. A variant, ExBEHRT+P, adds Med-BERT's prolonged-length-of-stay objective. The corpus comes from the Optum de-identified US EHR database, restricted to hospital encounters. Patients needed at least five visits with ICD-9 or ICD-10 diagnoses, which left 5.4 million individuals split 80/10/10 before training. Diagnoses were mapped to ICD-10 and truncated to three characters, leaving 1,916 diagnosis codes to predict. All models ran for 40 epochs on one 16 GB Tesla T4 GPU. ExBEHRT reached 64.2% masked-code precision, against 54.6% for BEHRT and 56.2% for Med-BERT retrained on the same data.
Fine-tuning used 437,903 cancer patients and 503,161 heart failure patients. ExBEHRT led every baseline on 6-month cancer mortality (AUROC 71.5%, APS 53.1%) and 12-month mortality (AUROC 74.3%). Heart failure readmission was harder for all models. ExBEHRT+P reached 56.8% AUROC, while XGBoost kept the highest average precision. The clustering experiment began from the general checkpoint, continued masked-language pretraining on cancer codes alone, and applied UMAP and HDBSCAN to CLS embeddings from 260,645 cancer patients. This placed 90% of them in 24 clusters, most labelled by a single cancer type.
The model is aimed at risk questions that depend on a patient's whole course of care: short-term mortality after a cancer diagnosis, readmission after heart failure, and grouping patients whose trajectories differ within one diagnosis. In the lymphoid leukaemia cluster, the embeddings separated a median-age-70 group from a median-age-5 group. These match chronic and acute lymphocytic leukaemia, although the three-character codes never distinguished them. Within pancreatic cancer, a second clustering pass isolated a subgroup with lower mortality. Per-token attributions let clinicians see which codes, visits and feature types drove a prediction.
ExBEHRT shows that the sequence-length cost of richer EHR features can be designed away, and that adding procedures, labs and lifestyle variables improves both pretraining and downstream prediction over diagnosis-only BEHRT. Its layout contrasts with CEHR-BERT, which concatenates concept domains into one sequence. The limits are clear from the paper. Every cohort comes from one commercial database, and no external institution was tested. Gains on heart failure readmission are small. BEHRT's reproduced pretraining precision on Optum fell short of its original report. No code or weights have been released: the only repositories the paper cites are the BEHRT and Med-BERT baselines. The authors built it on licensed Optum records, so it cannot be reproduced from public data.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.