Loading…
EHR foundation model that writes patient histories as token sequences carrying explicit visit and day-interval tokens, invertible back to OMOP tables.
No providers recorded yet. Browse all providers
Ask a synthetic electronic health record dataset a question like "which heart failure patients were readmitted within 30 days?" and most of them cannot answer, because the generator never recorded when anything happened. The point is sharp in this model's own ablation: a baseline built on CEHR-BERT's coarse week/month/year time tokens shrinks each patient's timeline enough that its synthetic heart-failure cohort shows a 100% 30-day readmission rate, against roughly 25% in the source data. The cohort definition survives; the interval it depends on does not.
CEHR-GPT's answer is to put elapsed time into the vocabulary rather than into the architecture. A patient history becomes a sentence: a demographic prompt of first-visit year, age, gender and race, then visit blocks bracketed by visit-start, visit-type, discharge and visit-end tokens, with artificial time tokens (ATT) between blocks that name the gap in days. Intervals past 1,080 days collapse to a single long-term token, and a distinct inpatient ATT marks the hours-to-days structure inside an admission. Because every field the OMOP Common Data Model needs is present as a token, a generated sequence can be decoded straight back into OMOP condition, drug, procedure and visit tables — the sequence representation is invertible, not lossy.
The model was developed at Columbia University by Pang et al. and posted as a preprint in February 2024, with collaborators at Washington University in St. Louis, NewYork-Presbyterian Hospital, the New York Genome Center and the OHDSI community. It sits alongside the Verily Multimodal EHR + Genomics Foundation Model as one of the few pretrained models that treat structured clinical records, rather than clinical text, as the sequence to be modeled.
The generative model is a standard GPT decoder stack: 16 transformer decoder layers, 8 attention heads, 128-dimensional embeddings and hidden units, a 512-token context window and 0.1 dropout, trained with next-token prediction for two epochs on two NVIDIA 2080 Ti GPUs at a learning rate of 2e-4. Training data came from the Columbia University Irving Medical Center–NewYork-Presbyterian OMOP database covering 3.7 million patients, of which roughly 2.3 million were used for training and 75,000 held out for privacy evaluation; patients averaged 16 visits and 148 tokens. One million synthetic patients were generated, 98% of whose sequences passed validation and converted to OMOP.
Evaluation ran at three levels — marginal concept prevalence, temporal co-occurrence, and predictive utility across five cohorts including heart failure and COPD readmission and 1-year Afib ischemic stroke risk. Nucleus sampling at top-p = 95% gave the best overall match; logistic regression on the synthetic hospitalization cohort reached 77.1 ROC-AUC against 75.3 on real data. Privacy metrics from the Yan et al. benchmarking framework — attribute inference 0.027, membership inference 0.127, identity disclosure 0.002 — fall well under the 0.333 threshold that framework treats as low risk.
The immediate use is giving researchers without a data use agreement something to work with: a shareable synthetic OMOP database supporting time-sensitive cohort construction, disease progression analysis and population estimation. Because the output is OMOP, it drops into the analytic stack an OHDSI site already runs. The repository's later pipelines extend the same sequence encoding into a reusable patient representation — embeddings for risk stratification and clustering via linear probing, and zero-shot time-to-event prediction in settings where labeled outcomes are scarce — though neither is evaluated in the 2024 paper.
CEHR-GPT established that the hard part of generative EHR modeling is the patient representation rather than the architecture, and its patient encoding has been extended by later work — most directly the same lab's CEHR-XGPT (September 2025), which scales the encoding up and folds representation learning, zero-shot prediction and generation into one model, and also a generative foundation model for menstrual health data. The maintained MIT-licensed repository has since grown well beyond the 2024 preprint, adding Medical Event Data Standard support and the linear-probing and zero-shot pipelines. Real constraints remain: sequences were capped at 20 to 512 tokens, which under-represents chronic-disease patients with long histories; the model over-generates prevalent concepts; resolution stops at whole days, limiting intensive-care use; and death is not yet a modeled event. No pretrained checkpoint is distributed, since the trained weights derive from protected clinical records — institutions run the released code against their own OMOP instance.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.