GARNET RNA Language Model

Rfam, the reference catalog of RNA families, draws its alignments from 14,451 UniProt reference genomes; SILVA covers only ribosomal RNA; RNAcentral is large but redundant. A language model trained on any of them inherits that ceiling. The Genome Taxonomy Database, by contrast, organizes over 400,000 bacterial and archaeal genomes — metagenome-assembled and single-cell amplified genomes included — into more than 85,000 species clusters, most from organisms nobody has cultured.

Researchers at the Innovative Genomics Institute mined that taxonomy to build GARNET (Gtdb Acquired RNa with Environmental Temperatures), then trained a GPT-style decoder on it. The paper does not give the model a proper name; the authors call it the 231-RNA LM, after the 231 RNA families it was pretrained on — 228 Rfam families found in bacteria and archaea, plus the 5S, 16S and 23S ribosomal RNAs. Its central finding is a tokenization result: representing RNA as overlapping nucleotide triplets, each token shifted one nucleotide from the last, encodes each of the four bases in sixteen different contexts and cuts validation perplexity from 3.66 under single-nucleotide tokens to 2.01 on the same 23S training set. The authors argue this captures base stacking, the dominant force in RNA folding, in a way single-nucleotide embeddings cannot without a far larger model.

GARNET's second contribution is phenotype. Every GTDB reference genome carries an optimal growth temperature — measured where TEMPURA or Gosha has one, predicted by TOME elsewhere — so the corpus can be sliced by thermophily. Fine-tuning on hyperthermophilic 23S rRNA and comparing generation probabilities against the pretrained baseline turns the model into a mutation ranker.

#Key Features

• Overlapping triplet tokenization: Tokens are three nucleotides wide with a one-nucleotide step, and outperformed single- and paired-nucleotide encodings at matched parameter count.
• Full-length rRNA generation from a seed: Given 100 nucleotides of the E. coli 23S or 16S 5' end, the pretrained model emits sequences that align to the Rfam covariance model across their entire length, scoring within the range of natural sequences under cmsearch.
• ∆∆logP mutation scoring: A candidate mutation is grafted into the wild-type E. coli 23S sequence, and its log-probability under the hyperthermophile-fine-tuned model is compared with its log-probability under the pretrained model, normalized to wild type — evaluating the mutation in full sequence context rather than position by position.
• Temperature-annotated corpus: Linking sequences to optimal growth temperatures makes phenotype-conditioned fine-tuning possible without structural or functional labels.
• Structure-aware companion model: A graph neural network trained on the 23S alignment conditioned on a fixed contact map gives an independent second opinion on candidates.

#Technical Details

The model is a modified nanoGPT: 18 attention layers, 6 heads, embedding dimension 300, a 384-token context window, rotary positional embeddings applied at every attention layer, RMS normalization in place of layer normalization, Flash attention, and AdamW — 19.48M parameters. Pretraining used 274M tokens with 31M held out for validation, reaching a validation perplexity of 1.61; fine-tuning on hyperthermophilic sequences brought it to 1.33. The companion graph neural network takes a binary contact map built from the k = 50 nearest neighbors of each nucleotide in the E. coli 23S structure (PDB 7K00) and reaches perplexity 1.751 with 1.09M parameters — an order of magnitude smaller, at the cost of a structural prior. When the authors tried GenerRNA, a byte-pair-encoded nanoGPT variant trained on RNAcentral, none of its three configurations produced full-length 23S-like sequences.

#Applications

The immediate application is ribosome engineering. Directed evolution of the E. coli ribosome for expanded genetic codes is limited by how poorly mutant subunits assemble, and a thermostabilized starting scaffold widens the tolerable mutational space. More generally, a pretrained generative model over 231 RNA families paired with a phenotype-annotated corpus lets RNA biologists nominate substitutions in riboswitches, T-box leaders and other structured RNAs, and generate synthetic family members for screening. Because the approach compares a fine-tuned model against its own pretrained baseline, it transfers to any phenotype for which a labeled subset of sequences can be assembled.

#Impact

The predictions were tested rather than left in silico. Purified 50S subunits carrying model-nominated substitutions were heat-treated at 65 °C and assayed for translation activity: a U2477C change in helix H89 left subunits roughly twice as active as wild type, and non-canonical base-pair changes in helix H68 also increased stability. Across the mutation groups tested, the models informed stabilizing changes in four of six cases, and the positions they flagged barely overlapped with those a frequency comparison of natural hyperthermophilic sequences would nominate. The scope is bounded: GARNET is built from bacterial and archaeal genomes only, with no eukaryotic, mitochondrial, chloroplast or viral sequences, and the authors describe these as deliberately small models. As RNA structure prediction continues to trail protein structure prediction — a gap the authors attribute to data rather than architecture, invoking ESM-2 as the protein counterexample — the database and the tokenization result are the parts most likely to carry forward.