GenEHR

A repeat blood panel a week after an abnormal result means active illness; the same panel a year later is a routine checkup. Only the gap separates them, yet sequence models over electronic health records usually blur it: a distinct token for every inter-visit gap at day-level resolution across a decade would need 3,650 of them, most far too rare to learn. GenEHR keeps the resolution and drops the vocabulary, writing each gap as five digits in a mixed-radix number system.

GenEHR is an autoregressive generative model of longitudinal patient health states, developed at Harvard Medical School with collaborators across five health systems. Pretraining pairs a next-event objective over tokenized clinical histories — diagnoses, medications, procedures, laboratory results, and static demographic and lifestyle variables — with a second objective predicting the digits of the gap to the next visit. The checkpoint supports conditional prediction, trajectory simulation and patient-level health-state representations directly, and is then adapted with a supervised time-to-event head for pan-cancer risk stratification as GenEHR-CancerRisk.

Because patient records cannot leave the institution holding them, a separate GenEHR is trained inside each of five secure data enclaves — Providence Health & Services, Mass General Brigham, the US Department of Veterans Affairs, UK Biobank and the NIH All of Us Research Program — sharing the tokenization scheme and the Chinchilla-tiered token-budget policy, but not a single architecture: the decoder family, width and vocabulary are chosen per cohort, with Llama-style decoders of differing sizes at Mass General Brigham, Providence, All of Us and UK Biobank and a GPT-Neo decoder at the VA. Each is pretrained on millions of patients and reused across tasks, not refit per prediction.

#Key Features

• RAdix Time Encoding (RATE): Each inter-visit interval becomes five digits with radices (7, 4, 6, 5, 5), giving place values of 1, 7, 28, 168 and 840 days. A 75-day gap is written (5, 2, 2, 0, 0), so day-level resolution survives on a handful of learned digit embeddings.
• Predicts when, not only what: Dedicated heads emit the gap digits from the visit token's hidden state, and the composed digit embedding goes back into the sequence, so generated events are conditioned on the elapsed time the model itself sampled.
• Gated low-rank adaptation for cancer risk: The backbone stays frozen while LoRA modules decompose into directions shared across cancer types plus a signed cancer-specific gate, letting rare cancers borrow evidence from common ones. A piecewise-exponential head then scores risk at horizons of 6, 12, 24, 36 and 60 months.
• Model-belief graphs: Contextual pointwise mutual information over generated trajectories yields a graph of events preceding a diagnosis, complemented by integrated-gradient and occlusion analyses.

#Technical Details

The backbone is a decoder-only transformer sized per cohort rather than shared, spanning two orders of magnitude: roughly 145M parameters at the VA, 108.3M at Mass General Brigham, 95.7M at Providence, 19.44M in All of Us and 1.18M in UK Biobank, with per-cohort vocabularies of 2,885, 6,147, 6,944, 4,694 and 1,281 tokens respectively. Diagnoses are three-character ICD-9/10-CM codes, medications RxNorm ingredients, procedures CPT or ICD-10-PCS codes, and labs LOINC codes kept only when outside the reference range and tagged high or low. After filtering, cohorts hold 13,187,301 patients at the VA, 6,884,118 at Providence, 1,999,232 at Mass General Brigham, 482,920 in UK Biobank and 312,181 in All of Us, with fixed held-out sets of up to one million patients. Next-visit event retrieval reaches AUROC 0.90–0.99 across cohorts and first occurrence of an ICD chapter 0.98–0.99, with expected calibration error of 0.07–0.14. Generated trajectories reproduce observed event prevalence at log-Pearson 0.93–0.99 and inter-visit gap distributions at Spearman 0.89–0.98. For risk over up to 17 cancer types, gated LoRA fine-tuning raises macro-AUROC at Mass General Brigham from 0.71–0.77 for direct scoring of the frozen model to 0.80–0.84. Outcome labels come from linked cancer registries rather than billing codes, and events in the three months before diagnosis are excluded.

#Applications

The intended use is nominating patients for risk-based screening where no population-wide programme exists. At an operating threshold of the 1,000 highest-risk patients and a 60-month horizon, standardized incidence ratios against demographically matched population incidence reach 34–48 for pancreatic cancer, 20–57 for lung and 6–22 for ovarian cancer across cohorts. Median lead time between the last event used for prediction and the registry diagnosis date runs 5.5–17.6 months at Mass General Brigham. The belief graphs serve a second use, surfacing associations such as biliary and pancreatic disorders before pancreatic cancer as hypotheses for epidemiological follow-up.

#Impact

GenEHR extends a line of EHR sequence models including representation-focused transformers such as CEHR-BERT and generative models like ETHOS, CoMET and Delphi; its contribution is fine-grained time handling combined with registry-confirmed supervised adaptation. The evaluation is retrospective across five cohorts on two continents, prospective testing is the stated next step, and performance falls under age-bracket and sex matching. It uses only structured records — no clinical notes or images — and inherits the unreliability of billing-derived diagnosis codes, which registry labels mitigate for outcomes but not inputs. The codebase is designated for GitHub release under an academic licence with restricted commercial use, and weights are obtained on request from each data-providing institution through the Sander lab at Harvard Medical School; the preprint awaits peer review.