Bacterial genome language model tokenizing whole genomes as ordered conserved elements; frozen embeddings beat Pfam baselines on 23 of 25 phenotypes.
No providers recorded yet. Browse all providers
Most newly discovered microbial species cannot be cultured and are known only from their genome sequences, which makes predicting phenotype from genome an increasingly load-bearing task. The standard representation for that task is a vector of thousands of binary indicators recording which conserved genomic elements — clusters of orthologous genes, or more often Pfam protein domains — are present. It is high-dimensional, hand-crafted, and throws away gene order entirely, even though position along a bacterial chromosome affects expression and gene clusters are conserved over evolution.
Whole-Genome Representation Learning (WGRL), from Cameron Dufault and Alan Moses at the University of Toronto, replaces that vector with a learned one. It defines a vocabulary of conserved genomic elements, tokenizes each bacterial genome as the ordered sequence of those elements, and pretrains a long-context genomic language model on hundreds of thousands of phylogenetically diverse genomes by self-supervision. The tokenization is what makes this tractable: nucleotide-level genomic language models top out in the hundreds of kilobases, well short of a 3.65 Mb bacterial genome, while element-level tokens compress a whole genome to a median of about 15,500 tokens. Prior gene-level models embedded each coding gene with a protein language model such as ESM-2 and stacked the results; WGRL instead learns its token embeddings over a compact shared vocabulary. The work was presented at the LMRL workshop at ICLR 2025.
The model has 16 layers, a hidden size of 768 and 69,532,906 parameters. Training data came from all 371,175 bacterial genome assemblies in RefSeq as of August 2024; roughly 280 million unique protein sequences were scanned with HMMER3 against Pfam-A v35.0, keeping domain hits with score above 10. Genomes were split by phylogeny — 5% of genera held out for test, then 5% of remaining species and assemblies — before redundancy filtering and upsampling produced a training set of 535,967 genomes. Only tokens appearing in genomes from at least 1,000 of the 3,459 represented genera entered the vocabulary. Pretraining ran 4 epochs at batch size 32 with AdamW and a cosine schedule, taking 8 days on a single NVIDIA RTX A6000.
Evaluation used 15,926 BacDive-labelled genomes across 20 binary and 5 continuous phenotypes spanning morphology, growth conditions, metabolism and environment. Final-layer representations were averaged across the sequence and fed to a weighted k-NN classifier or regressor under cosine distance with 5-fold cross-validation. At k = 15, WGRL beat Pfam presence/absence on 23 of 25 phenotypes — indole test MCC 0.527 versus 0.424, binary cell length 0.601 versus 0.523, optimal salinity Spearman 0.630 versus 0.605 — including phenotypes such as pH and salinity preference that carry little phylogenetic signal.
The target users are microbial genomics groups doing genotype-to-phenotype inference on organisms that cannot be cultured: assigning oxygen tolerance, motility, sporulation, morphology or growth optima to metagenome-assembled genomes, or screening assembly collections for strains with a phenotype of interest. Because the representation is a fixed-length vector produced without task-specific training, it also drops into existing supervised pipelines wherever domain presence/absence vectors are used today.
WGRL is a workshop paper rather than a peer-reviewed journal or conference publication, and its practical availability is limited: the Apache-2.0 repository provides the dataset construction and training code, but its README describes the project as a work in progress and lists downloadable pretrained models and the k-NN evaluation code as not yet added, so the trained checkpoints cannot currently be obtained. The tokenization imposes a structural limit as well — element-level tokens cannot express nucleotide-level determinants, so phenotypes driven by point mutations, GC content or codon bias, including several antibiotic resistance phenotypes, fall outside its reach. What it does establish is that gene order and long-range element context carry recoverable phenotypic signal that the field's default bag-of-domains representation discards.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.