UKBioBERT

Genomic language models are almost always pretrained on the reference genome — a single consensus sequence that, by construction, contains none of the variation that distinguishes one person from another. Sequence-to-function models, meanwhile, predict expression from sequence but are usually trained without individual-level genotypes. The gap between them is exactly where personalised functional genomics lives: how much of an individual's gene expression is predictable from their own genome, and which variants drive the difference.

UKBioBERT, from Hongyu Zhao's group at Yale University with collaborators at the Broad Institute and Massachusetts General Hospital, addresses that gap by putting human variation into pretraining. Starting from the DNABERT-2 checkpoint, the model is continue-pretrained with masked language modelling on reference sequences edited to carry real genetic variants — substitutions, insertions, and deletions — drawn from the European-ancestry cohort of the UK Biobank. The resulting encoder produces DNA embeddings that are variant-aware rather than reference-only.

The release is a family built around that encoder. UKBioBERT itself supplies embeddings; UKBioFormer and UKBioZoi fuse those embeddings into the Enformer and Borzoi sequence-to-function architectures respectively, using parameter-efficient fine-tuning over the frozen backbones, at 230.7M and 170.7M parameters. Together they cover the pipeline from representation learning through individual-level expression prediction to in-silico mutagenesis and eQTL inference.

#Key Features

• Variant-aware pretraining: reference sequences are edited in place with UK Biobank variants through explicit replace, insert, and delete operations, then used for masked language modelling — 13 million variants and their paired sequences, split 80/10/10 for training, validation, and checkpoint selection.
• Objective chosen by comparison, not assumption: masked language modelling was selected after benchmarking against contrastive pretraining and against supervised prediction of linkage disequilibrium scores, both of which produced weaker gene representations.
• Sequence-to-function fusion: UKBioFormer and UKBioZoi concatenate UKBioBERT embeddings with Enformer or Borzoi sequence embeddings through a fusion layer, training only a small set of adapted parameters and leaving the pretrained backbones frozen.
• Interpretability tooling: in-silico mutagenesis on UKBioFormer recovers the direction of expression effects for individual variants, and the same machinery is used to infer eQTLs without matched genotype-expression data.
• Flexible context: the encoder accepts sequences of arbitrary length subject to memory; a single H100 accommodates a 100 kb input.

#Technical Details

On a gene-function clustering benchmark scoring embeddings by the average of normalised mutual information, adjusted Rand index, and average silhouette width, UKBioBERT reached 0.423, ahead of GPN (0.333), a variant-pretrained HyenaDNA control (0.330), HyenaDNA itself (0.320), DNABERT-2 (0.319), Nucleotide Transformer (0.299), Borzoi (0.160), Enformer (0.146), and Evo 2 (0.062) — the last several being models orders of magnitude larger. Data scaling experiments varying the fraction of variants included (10%, 50%, 100%) showed performance rising monotonically with variant coverage, isolating the contribution of the variant corpus itself.

Downstream, adding UKBioBERT embeddings to EPInformer improved gene expression prediction in the K562, GM12878, and HepG2 cell lines under EPInformer's own 12-fold cross-validation splits. For individual-level prediction, evaluated on 670 GTEx samples with paired RNA-seq and whole-genome sequencing, UKBioFormer outperformed personalised Enformer on 63.3% of well-predicted genes (PCC above 0.6) while using less runtime and GPU memory; UKBioZoi trailed slightly but trained faster still. On eQTL direction inference, over 70% of predicted eQTLs carried the correct sign, with accuracy higher for variants whose effect sizes carry larger standard error.

#Applications

The models serve population and functional genomics: quantifying which genes have expression predictable from sequence at all, predicting individual-level expression in cohorts with genotype but no transcriptome, prioritising regulatory variants, and inferring eQTL direction and magnitude where matched expression data are unavailable. The embeddings also transfer to gene-function annotation. Because Enformer and Borzoi backbones are handled interchangeably, the fusion recipe generalises to other sequence-to-function models.

#Impact

The work's clearest finding is that adding real human variation to pretraining — not more parameters — is what improved these representations, with a 0.42 clustering score from a BERT-scale encoder beating billion-parameter genome models trained on reference sequence alone. It is equally clear about where the approach breaks: for a model trained on European-ancestry individuals and tested on African American individuals, UKBioBERT embeddings alone predicted poorly, with negative correlations for some genes, and only the sequence-to-function fusions recovered usable accuracy. Group-level training over enhancer-sharing or pathway-sharing gene sets did not reliably help. The most consequential practical constraint is governance: because pretraining used individual-level UK Biobank genetic data, the weights cannot be redistributed and are returned to UK Biobank, so obtaining them requires going through UK Biobank's own controlled-access process. Code is public, but the repository carries no licence file. The work has since been peer reviewed and published in npj Artificial Intelligence.