CEHR-BERT

Two patients carry the same diagnosis, the same drug and the same follow-up procedure, in the same order. In one chart those events span six weeks; in the other, six years. Treat a medical history as a document and feed the concept codes to BERT as words, and the two patients become the same sentence — the distinction between an acute episode and a slow chronic course disappears before the model sees anything. Earlier BERT adaptations for structured records shared this gap: some marked visit boundaries but modeled only diagnosis codes, others added concept domains but dropped temporal signal.

CEHR-BERT restores the calendar by writing it into the vocabulary. Between two neighboring visits it inserts an artificial time token (ATT) naming the gap — W0 through W3 for intervals under 28 days, M1 through M11 up to a year, and a single LT long-term token beyond that — and brackets each visit with VS and VE tokens so the encoder can see where one encounter ends and the next begins. Absolute time and patient age are separately encoded as continuous features through a time2vec Fourier decomposition, concatenated with the concept embedding and passed through a fully connected layer to form a temporal concept embedding. Because BERT's next-sentence prediction objective is meaningless when the whole patient history is one sentence, the authors replaced it with visit type prediction: a decoder layer reconstructs masked inpatient/outpatient/emergency labels, forcing the encoder to learn that different care settings draw on different concept distributions.

The model was built by Pang et al. at Columbia University Irving Medical Center with a collaborator at Geisinger, and presented at Machine Learning for Health 2021. Its patient representation is the direct ancestor of the same lab's generative successor CEHR-GPT, which replaces the coarse week/month/year tokens with day-level intervals.

#Key Features

• Artificial time tokens: Interval tokens between visits let elapsed time be attended to by the same mechanism as diagnoses, drugs and procedures.
• Explicit visit boundaries: VS and VE tokens define visit segments; removing them in ablation dropped performance to the level of a plain separator token.
• Continuous time and age embeddings: A time2vec transformation encodes absolute date and patient age, capturing seasonal and age-dependent conditions alike.
• Visit type prediction: A second learning objective built for EHR data, in place of BERT's next-sentence prediction.
• Label-efficient transfer: Fine-tuning on 5% of the training data outperformed comparison models trained on the full set.

#Technical Details

The encoder is small by language-model standards — 9.1 million trainable parameters across 5 transformer layers and 8 attention heads, with 128-dimensional embeddings and hidden units, 0.1 dropout, and a 300-token context window chosen over BERT's usual 512 because 300 tokens already cover more than 90% of patient histories. Pretraining ran for 5 epochs with Adam, batch size 32 and a 2e-4 learning rate under cosine annealing, at 8.5 hours per epoch on two RTX 2080 Ti GPUs. The corpus was the Columbia University Irving Medical Center–NewYork-Presbyterian OMOP instance: 2.4 million patients contributing 184.7 million clinical data points across the condition, procedure and medication domains, reaching back to the early 1980s.

Evaluation used 4-fold cross-validation with a 75:10:15 split on four cohorts — 2-year hospitalization risk (590,578 patients), 1-year mortality after discharge home (207,919), lifetime heart failure onset in type 2 diabetes (114,564), and 30-day heart failure readmission (97,758). CEHR-BERT led on both ROC-AUC and PR-AUC in all four, reaching 80.7 AUC on the diabetes-to-heart-failure task, the only model above 80, with PR-AUC of 0.323 against 0.274 for the next best; mortality after discharge home scored 94.6 AUC, hospitalization 75.9 and readmission 66.3. Ablations stripping the time and age embeddings, the visit boundary tokens or the visit type objective each cost accuracy, and a variant without pretraining trailed the full model on every task.

#Applications

Any site running an OMOP Common Data Model warehouse can pretrain the model on its own records and fine-tune it for cohort-level risk questions — readmission, mortality after discharge, progression from one condition to another. Label efficiency matters most for phenotypes where chart review is the bottleneck and only a few thousand adjudicated cases exist. The released framework covers the path from OMOP tables to a fine-tuned classifier.

#Impact

CEHR-BERT made the case that temporal structure belongs in the patient representation rather than in the architecture, and that argument carried into the lab's later generative work. The MIT-licensed codebase has been maintained well past the paper, rewritten onto the HuggingFace transformers stack, extended to the Medical Event Data Standard alongside OMOP, and given a linear-probing path over frozen patient embeddings. Real limits remain: the 300-token window truncates the longest histories, an effect the time tokens themselves worsen by inflating sequence length; laboratory values were excluded because continuous measurements need a different embedding strategy; and both pretraining and evaluation come from a single health system. No pretrained weights are distributed — the checkpoint derives from protected clinical records, so institutions run the code against their own data.