MethylQUEEN

MethylQUEEN — Methylation Language Model based on Quintuple Bidirectional Transformer — is a DNA foundation model that treats 5-methylcytosine as a letter of the genetic alphabet rather than as an annotation layered on top of one. Bisulfite sequencing resolves methylation at single-base resolution, but most computational pipelines immediately collapse the reads into per-site beta values, discarding the pairing between methylation marks that co-occur on the same DNA fragment. That read-level information is exactly what matters for liquid biopsy, where a plasma sample is a mixture of fragments from many tissues and the question is which fragment came from where.

The model was developed by Mingyang Li, Ruichu Gu, Chengqi Yi and colleagues at Peking University and the AI for Science Institute in Beijing, and posted to bioRxiv in December 2024. Its central design choice is a five-letter representation: sequences are written over A, C, G, T and M, where M marks an unconverted — that is, methylated — cytosine. Because the modification lives in the vocabulary, pretraining sees it directly, and the tokenizer can be extended to other marks such as m6A without redesigning the architecture. This distinguishes it from methylation-aware transformers such as MuLan-Methyl, which classify annotated sites, and from sequence-only genomic models such as DNABERT and Evo that carry no epigenetic signal at all.

#Key Features

• Quintuple alphabet: Methylation state is encoded in the sequence itself, so a byte-pair-encoding tokenizer trained over the methylated genome produces "methyl-tokens" containing CG or M that cluster into a distinct region of embedding space after pretraining.
• Methyl-token-prone pretraining: A second pretraining stage masks methyl-tokens almost exclusively while leaving ordinary tokens unmasked, forcing the model to reconstruct methylation state from sequence context rather than memorizing base composition.
• Read-level deconvolution: The methylation-to-components task assigns individual reads to a tissue or disease component, supporting mixture estimation in plasma cell-free DNA instead of averaging signal across a region.
• Methylation imputation: Masked methyl-tokens in PacBio HiFi libraries are recovered with AUROC between 0.925 and 0.931, filling in ambiguous calls from low-confidence regions.
• Attention-based site attribution: Summed attention across heads and layers highlights the specific CpG sites driving a classification, recovering a c-Myc motif at an MSX1 promoter site in a lymphoma-versus-normal comparison.

#Technical Details

MethylQUEEN builds on the MosaicBERT encoder, with Attention with Linear Biases in place of positional embeddings, FlashAttention, low-precision layer normalization, and GeGLU activations; LoRA is used for parameter-efficient fine-tuning. The released checkpoint is a 16-layer encoder with 1,024 hidden dimensions and 16 attention heads over a byte-pair vocabulary of 8,192 tokens. Pretraining data came from whole-genome bisulfite sequencing in the ENCODE and NIH Roadmap Epigenomics projects — more than 500 samples spanning over 30 human tissues — aligned to GRCh38 and assembled from short reads into fragments averaging over 600 bp, yielding 568,063,338 high-quality methylation-encoded sequences. Training ran for 300,000 steps of standard masked language modeling at a 0.15 masking rate, then 280,000 further steps with methyl-tokens masked at roughly 97%, on eight NVIDIA A800 GPUs. On component-assignment benchmarks the model improves AUC by up to 6.03% over MethylBERT, DISMIR and AutoGluon baselines, and a single CpG site per read is enough for effective classification.

#Applications

The immediate use is cell-free DNA analysis: tracing plasma fragments back to their tissue of origin, deconvolving bulk tissue mixtures, and estimating tumor burden. Applied to targeted bisulfite libraries from hepatocellular carcinoma patients, the predicted tumor burden for stage I disease was significantly elevated relative to normal plasma, the stage where liquid biopsy is hardest and most valuable. The methylation-to-expression head predicts transcript abundance from the methylation state of a 4 kb window around a transcription start site, which is useful when RNA is unavailable but archival DNA is, and the attention maps give epigenetics groups a way to nominate candidate regulatory CpG sites for follow-up.

#Impact

MethylQUEEN is a demonstration that epigenetic marks belong in pretraining rather than in a downstream feature table, and that the vocabulary is a natural place to put them. The approach generalizes: any modification detectable at base resolution can become another letter. A pretrained checkpoint is published on Hugging Face under an MIT license by the first author, but no training or inference code repository accompanies it, so reproducing the pipeline requires rebuilding the data processing from the Methods. The manuscript remains a preprint awaiting peer review, the preprint itself carries a CC BY-NC-ND license, and evaluation is confined to human data from two consortium projects — cross-species and prospective clinical performance are untested.