DNA foundation model encoding the genome as probabilistic allele frequencies from the 1000 Genomes Project. 86M parameters, single-nucleotide tokens.
No providers recorded yet. Browse all providers
MutBERT is a genomic foundation model from the ai4nucleome group at the Hong Kong University of Science and Technology (Guangzhou) that changes what a DNA language model reads. Conventional genomic language models consume a single reference genome as a string of discrete nucleotide tokens, discarding everything known about how the genome varies between individuals. That also makes population-scale resources awkward to exploit: single nucleotide polymorphisms are rare, so training a masked language model on thousands of near-identical individual genomes spends almost all of its compute re-learning identical sequence.
MutBERT instead represents each genomic position as a probability distribution over the four nucleotides, computed from observed allele frequencies. Every input position is a four-element probability vector rather than a token index, so the model sees both the reference base and how variable that site is across the population in one representation, and pretraining can focus the masking budget on informative variable sites.
Three checkpoints were released, differing only in the source of the probability distributions: MutBERT, trained on allele frequencies from the 1000 Genomes Project; MutBERT-Human-Ref (called MutBERT-Ref in the paper), trained on the hg38 reference with probability 1 assigned to the reference allele; and MutBERT-Multi, trained on probabilities from a 100-vertebrate multiz whole-genome alignment anchored to the human genome, following the approach of GPN-MSA. The paper is a preprint and has not been peer reviewed.
transformers AutoModel interface. The GitHub repository carries no license file, and the preprint is published under a no-derivatives licence, so the weights are the only cleanly reusable part of the release.MutBERT is a 12-layer BERT-style encoder with GELU activations, 768-dimensional hidden states, and Flash Attention, totaling 86 million learnable parameters. Population-level probabilities were computed from 3,202 high-coverage genomes spanning 27 populations in the 1000 Genomes Project, discarding alleles with frequency below 2%. All three variants were pretrained for 120,000 steps at batch size 2,048 and sequence length 512 with AdamW and a cosine schedule, corresponding to roughly 126 billion tokens seen and about 40 epochs over the unique token set.
Evaluation covered 24 human genomic datasets: transcription factor binding site prediction from the GUE benchmark, 18 datasets from the Nucleotide Transformer downstream benchmark, and an eQTL variant effect task. MutBERT reached the highest average Matthews correlation coefficient of 68.29 on the TFBS tasks, marginally ahead of DNABERT-2, and an average MCC of 66.36 across the 23 MCC-scored tasks — third overall, behind only the 498-million-parameter and 2.5-billion-parameter multi-species Nucleotide Transformer variants. Restricting the comparison to models of comparable size or training scale, MutBERT was best on 13 of 23 tasks. On the eQTL task it reached 0.615 average AUROC, slightly behind MutBERT-Multi. The ablation is informative: MutBERT outperforms both MutBERT-Human-Ref and MutBERT-Multi on average, indicating the gain comes from population allele frequencies specifically rather than from the probabilistic input format alone.
The released checkpoints serve as frozen feature extractors or fine-tuning starting points for human genomic prediction tasks — transcription factor binding, epigenetic marks, promoter and enhancer classification, splice site prediction, and expression QTL variant effect scoring. Because embeddings come from a standard transformers call, the models drop into existing genomics pipelines without custom infrastructure, and the 86-million-parameter footprint makes fine-tuning practical on a single GPU. The design is most directly useful to groups working with population-scale variation data, where the probabilistic input naturally encodes cohort-level allele frequencies.
MutBERT's main contribution is methodological: it folds biobank-scale resequencing data into a genomic foundation model as a change of input representation rather than as more training sequence, sidestepping the redundancy that makes naive population-scale pretraining inefficient. Competitive accuracy at 86 million parameters against models an order of magnitude larger supports the broader argument that representation choices matter more than scale for DNA models. Adoption is still early — the repository has modest engagement and HuggingFace download counts are low — and the evaluation is confined to human tasks. The approach has since been carried further: reference-based and variation-based pig genome checkpoints now sit alongside the original three under the same HuggingFace organization.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.