Sparse autoencoder on genomic language model activations that finds conserved microbial intergenic elements, from operator sites to structured RNAs.
No providers recorded yet. Browse all providers
Most of what is known about microbial noncoding DNA was found by searching for something already known: a motif, a covariance model, an Rfam family. Undescribed elements have no query. A riboswitch that has drifted beyond its family's profile, or an operator whose inverted repeats differ from genus to genus, is invisible to homology search even when it sits beside the same gene across hundreds of genomes.
gLM2-SAE approaches this from the other direction. It is a sparse autoencoder that decomposes the internal representations of gLM2, a genomic language model that reads protein-coding genes as amino acids and intergenic regions as nucleotides in their native order. Because gLM2 sees each intergenic position with its neighboring genes in context, the autoencoder's 16,384 sparse features can pick out recurring nucleotide patterns without any predefined motif, and each feature can then be traced to the genes, protein domains and RNA families it repeatedly sits beside. It was developed by Tatta Bio with collaborators at MIT and described in a September 2026 bioRxiv preprint by Zulaybar, Tranzillo, Silverstein, Hwang and Cornman.
The features are indexed in SeqHub, the multimodal genomic search platform that succeeded Gaia, so a researcher can start from either an unexplained intergenic pattern or a protein of interest and move between the two. Like InterPLM for protein language models, it is an interpretability layer over a pretrained backbone rather than a new backbone; unlike protein-side SAEs, it is aimed at noncoding DNA.
gLM2-SAE is a BatchTopK sparse autoencoder with 16,384 latent features and K = 8, so on average eight features are active at each position. It reads residual-stream activations after transformer block 24 of the frozen 33-layer gLM2-650M (hidden dimension 1,280), a twelvefold expansion, and has about 42 million parameters. Training windows of 4,096 tokens were drawn from the OpenGenome (OG) dataset, centered on 50%-identity cluster representatives to limit over-represented isolate genomes. The SAE was trained on 10^9 intergenic positions with Adam (learning rate 7 × 10^-5, batch 32,768) on eight H100 GPUs, using code adapted from SAELens.
Applied across OG's roughly 300 million intergenic regions, firing positions within 3 nt were merged into spans and spans shorter than 8 nt dropped, leaving 3,656 features seen in at least 50 contigs. To test whether spans reflect real sequence signal, STREME motifs were learned on 2,234 features' top regions and scored on genera held out from motif fitting: the median AUROC was 0.893, and 48% of features exceeded 0.9, against 0.429 for random intergenic regions from the same contigs. Across genus-dereplicated contigs, 605 features (16.5%) co-occurred with the same Pfam domain in more than half of cases, and 290 (7.9%) with the same Rfam family.
The model is a discovery tool for microbiologists and RNA biologists mining the noncoding fraction of bacterial and archaeal genomes. The preprint's worked examples show the pattern: feature #790 marks inverted-repeat operator candidates upstream of the YoaS-YozG operon, feature #7917 recovers selenocysteine insertion sequence (SECIS) elements that the SECIS-3 and SECIS-4 Rfam families miss, features #14380 and #6752 pick out divergent candidate riboswitches beside membrane transporters, and feature #13341 marks a conserved RNA near ribosomal-protein genes with covarying secondary structure and no Rfam match.
gLM2-SAE extends sparse-autoencoder interpretability from protein language models to the intergenic DNA of a mixed-modality genomic model, turning a genomic language model into a source of testable hypotheses about regulatory and RNA elements. It is a preprint without peer review, its structured-RNA candidates rest on computational covariation evidence rather than wet-lab validation, and it covers intergenic positions only. The weights are on HuggingFace and inference code is on GitHub, while the indexed spans and annotations are served through SeqHub, which is free for academic users signing up with an academic email and sold as a subscription for commercial use.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.