MethylGPT

An epigenetic clock built the conventional way is a penalized linear regression over a few hundred hand-picked CpG probes. Each site enters as an independent predictor, so a probe the array does not carry is an unrecoverable hole, a tissue the clock was never fitted on is out of distribution, and every new endpoint means another regression from scratch.

MethylGPT replaces that per-task modelling with one pretrained transformer over the human methylome. Its central design choice is tokenization: every CpG probe gets an integer identity token, that token and the site's methylation value are embedded separately, and the two embeddings are summed element-wise. A site is represented by both which CpG it is and how methylated it is, and self-attention relates distant CpGs rather than scoring them in isolation. Masked-value prediction over that representation is one objective from which imputation, sample embedding, and downstream prediction all follow.

The model came from Kejun Ying and colleagues in the Gladyshev lab at Harvard Medical School and Brigham and Women's Hospital, with Bo Wang's group at the University of Toronto and the Vector Institute. Its backbone is adapted from scGPT: the formulation that treats a single-cell expression profile as a bag of gene tokens carries over to a methylation profile as a bag of CpG tokens. It sits alongside CpGPT and MethylAI among methylation foundation models, and differs from both in working directly on array beta values rather than deriving methylation from genomic sequence.

#Key Features

• Missing-data tolerance: Recovering absent probes is the pretraining objective itself, not a bolt-on. Age prediction stays stable with up to 70% of CpG sites masked, where ElasticNet and MLP baselines degrade.
• Unsupervised biological organization: Zero-shot sample embeddings cluster by tissue — whole blood, brain, liver, skin — and separate male from female samples, with batch effects weaker than in UMAPs of the raw beta values.
• A CpG vocabulary chosen for cross-platform use: The 49,156 sites are those tied to more than five EWAS catalog traits or present in over 95% of the pretraining corpus, so Illumina 27k, 450k, and EPIC profiles map onto one shared token space.
• Attention as an interpretability handle: Attention concentrates on a sparse set of CpGs differing between young and old samples; sites attended to in older samples enrich for oxidative stress and amino acid metabolism, those in younger samples for development.
• Three released checkpoints: base, medium, and large share a six-block, four-head layout and differ only in embedding width — 64, 128, and 256 dimensions, or 3M, 7M, and 15M parameters.

#Technical Details

Pretraining drew 226,555 human array methylation profiles from 5,281 datasets in the EWAS Data Hub and Clockbase; after dropping entries missing more than 40% of sites and deduplicating, 154,063 samples across more than 20 tissue types remained, yielding 7.6 billion CpG tokens. Two losses were optimized together: masked prediction of the methylation value at 30% of randomly masked sites, and reconstruction of the whole profile from the [CLS] token embedding alone. Training used AdamW at a learning rate of 0.001 for 10 epochs with FlashAttention over the 49,157-token sequence, reaching a test mean squared error of 0.014, a mean absolute error of 0.074, and a Pearson correlation of 0.929 against measured values.

Fine-tuned with a ResNet1D head on 11,453 samples spanning ages 0 to 100, MethylGPT predicted chronological age to a median absolute error of 4.45 years, ahead of ElasticNet, the AltumAge MLP, and Horvath's skin and blood clock on the same splits. The same head fine-tuned on 18,859 Generation Scotland participants predicted 10-year risk across 60 conditions and overall mortality at an AUC of 0.74 on validation and 0.72 on test.

#Applications

One checkpoint covers work that previously needed several purpose-built models: filling gaps in archival array data, harmonizing cohorts run on different Illumina platforms, embedding samples for clustering and quality control, and seeding new methylation-derived endpoints. On iPSC reprogramming time courses the embeddings traced a progressive shift toward a younger methylation state, agreeing with Horvath's clock and GrimAge on the direction of rejuvenation. On intervention data pooled from six GEO studies the disease head ranked smoking cessation as the strongest protective effect on 10-year mortality.

#Impact

MethylGPT is the argument that array methylation, long the province of penalized linear models, supports the same pretrain-then-adapt workflow that reshaped protein and single-cell analysis. The code is Apache-2.0 on GitHub with a methylgpt PyPI package, tutorials for each documented workflow, and weights for all three sizes. Real caveats remain: the work is a preprint awaiting peer review, the checkpoints ship through Google Drive folders with no versioning, checksums, or license of their own, and the clinical evaluation rests on a single Scottish cohort. The model is also confined to the array probe space it trained on — whole-genome bisulfite and single-cell methylation stay outside its vocabulary, which the authors name as future work.