Loading…
DNA language model for unassembled metagenomic reads, pretrained to recover coding fraction and reading frame from 60-300 bp fragments.
No providers recorded yet. Browse all providers
Most metagenomic analysis starts by assembling reads into contigs and genes. That step discards information: assembly fails in low-coverage or highly diverse communities, and in extreme environments the reads that survive it are a small minority of what was sequenced.
REMME (Read EMbedder for Metagenomic Exploration), from Yana Bromberg's group at Emory University, is a pretrained DNA language model whose unit of analysis is a single short read rather than a chromosome or a gene. Where genome-scale DNA models such as Evo and the Nucleotide Transformer learn from long assembled sequence, this one learns from 60–300 bp fragments that carry no surrounding context at all — a regime where the usual tricks of long-range attention have nothing to attend to.
The design response is to make the pretraining objectives do more work. Alongside masked-token prediction, the model is trained to estimate what fraction of a read is protein-coding and in which reading frame, which forces it to recover biological structure rather than settling for nucleotide composition statistics. It is the backbone released with REBEAN, the enzyme annotator fine-tuned on top of it. The work was posted to bioRxiv in December 2024 and published in Nucleic Acids Research in September 2025.
The model is an encoder-only transformer with six layers and eight attention heads. It embeds overlapping nucleotide triplets at stride one into 128 dimensions alongside a 128-dimensional positional embedding, for 1,662,177 trainable parameters in total.
Pretraining used 72.9 million reads sampled from 1,496 marine prokaryotic assemblies in MGnify, clustered at 80% identity down to 53.6 million representatives. Three heads share a summed loss: a masked-token decoder, a regression head for coding fraction, and a four-way reading-frame classifier. The trained model reached over 98.5% masked-token accuracy, correlated with ground-truth coding fraction at Pearson r = 0.73, and separated coding from non-transcribed reads at 88.5% accuracy. The value of that pretraining is measurable downstream: training the fine-tuned annotator's architecture from random initialization instead plateaued at 30% accuracy against 80.6% from the pretrained start.
The direct use is as an embedding source for read-level metagenomic tasks — supplying representations for classifiers, clustering, or similarity search over raw sequencing reads where no assembly and no reference database are available. Its released fine-tune, REBEAN, demonstrates the pattern: a three-layer dense classifier on top of the frozen representation is enough to reach usable enzyme-class annotation. The checkpoint ships alongside the fine-tuned one in the same package, so the encoder can be loaded on its own for new downstream heads.
This extends language modeling into a regime that had largely been skipped: the short, context-free sequencing read. The contribution is less the architecture, which is deliberately small and conventional, than the demonstration that auxiliary objectives grounded in gene structure can substitute for the long-range context these models normally depend on. The published artifacts include the checkpoint, training scripts, and a Figshare archive under CC-BY-4.0. Limitations are inherited from the pretraining corpus: it is drawn entirely from marine prokaryotic assemblies, so performance on eukaryotic, viral, or host-associated reads is not characterized, and the paper reporting every number is licensed for non-commercial reuse only.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.