CEHR-XGPT

A model that compresses "three days later" and "three months later" into the same token has already discarded what most clinical questions turn on. Earlier EHR sequence models coarsened irregular timing into age tokens that advance only on a birthday, or buckets spanning 6–12 hours: fine for forecasting the next diagnosis, fatal for rebuilding a 30-day readmission cohort from the output.

CEHR-XGPT keeps the interval in the vocabulary and then teaches the model what it means. Each record becomes a sequence: a demographic prompt of start year, age, gender and race, then visit blocks bracketed by visit-start and visit-end tokens, separated by an artificial time token (ATT) giving the gap in days. Two auxiliary objectives attach at those ATT positions. Time Decomposition splits each time-token embedding into year, month and day sub-embeddings and predicts all three; Time-to-Event fits a Gamma distribution over the true elapsed days through a small feed-forward head. Without them, time tokens carry little beyond relative position.

The result is one checkpoint covering three jobs usually split across three models: patient feature representation, zero-shot prediction, and synthetic record generation. Pang et al. built it at Columbia University's Department of Biomedical Informatics with collaborators in the OHDSI community and at NewYork-Presbyterian Hospital, posting it as a preprint in September 2025. It succeeds CEHR-GPT, which introduced the time-token representation for synthetic generation alone.

#Key Features

• Three capabilities, one checkpoint: Frozen patient embeddings, prompt-driven zero-shot prediction and synthetic timeline generation all come from the same pretrained model, with no retraining between them.
• Time Decomposition objective: Predicting a gap's year, month and day components separately lets long intervals share learned structure with short ones instead of being memorized as unrelated symbols.
• Time-to-Event objective: A feed-forward head at every time token parameterizes a Gamma distribution over the elapsed days, training on how long gaps last rather than only on which token comes next.
• Prediction by simulation: The model rolls out 50 simulated futures from a patient prompt, and the outcome probability is the fraction in which the event lands inside the window.
• Transfer by vocabulary expansion: Adapting to an outside health system means adding its unseen OMOP concept IDs to the tokenizer and fine-tuning, not pretraining from scratch.

#Technical Details

The backbone is GPT-2 with 16 decoder blocks, 12 attention heads, 768-dimensional embeddings and hidden units, a 4,096-token context window and 0.1 dropout; learned positional embeddings are dropped entirely, since the time tokens already carry ordering. Pretraining ran 10 epochs on a single NVIDIA A6000 with sample packing, batches of up to 16,384 tokens and a learning rate of 0.001. The corpus is a subset of the Columbia University Irving Medical Center–NewYork-Presbyterian OMOP database covering conditions, medications and procedures: roughly 2.6 million patients for training and a further 1 million held out for evaluation, averaging 16 visits and 117 concepts each.

Six Columbia cohorts anchor the evaluation. Zero-shot and unmodified, the model reaches 66.8 AUROC on 30-day heart-failure readmission, 90.9 on one-year CAD CABG and 78.9 on lifetime heart failure following type 2 diabetes, beating count-based baselines on two of the three. On the external EHRSHOT benchmark of 6,739 Stanford records — reached by adding over 5,000 Stanford-specific OMOP concept IDs to the tokenizer — it posts the best reported New Diagnosis results. An ablation removing only the two time objectives matches it on in-distribution supervised tasks but falls behind on zero-shot prediction, treatment-pathway fidelity and external validation.

#Applications

The intended user is a hospital informatics group already running an OMOP instance. Linear probing turns any cohort into embeddings for risk stratification, patient clustering and propensity-score matching without a separate model per question, and zero-shot prediction answers outcomes with no labeled cohort yet — emerging diseases, low-resource sites, definitions that change faster than a labeling pipeline. The generative path yields a synthetic OMOP database, 3.7 million records in the authors' run, analyzable with ordinary OHDSI tooling and scoring below the 0.333 risk threshold on all four Yan et al. privacy attacks.

#Impact

CEHR-XGPT argues that multi-task breadth in an EHR foundation model comes from the patient representation rather than a bigger backbone: the same architecture stripped of the two time objectives underestimates hypertension and diabetes prevalence in treatment-pathway replication (0.26% and 0.10% against real rates of 0.45% and 0.18%) and loses ground on every out-of-distribution test. Real limits remain. MOTOR, an EHR model pretrained on time-to-event objectives, still leads on linear probing across all six Columbia cohorts, a gap the authors trace to their own next-token pretraining favoring sequence modeling over embedding quality. Coverage spans conditions, medications, procedures and visits but not measurements, labs or observations, and the synthetic dataset carries no death or measurement domain, which forced two cohorts out of that evaluation. No pretrained checkpoint is distributed, since the weights derive from protected patient records; the MIT-licensed repository is the deliverable, and institutions train against their own OMOP data.