Non-coding RNA language model using masked discrete diffusion to unify sequence generation with representation learning, trained on 30M ncRNAs.
No providers recorded yet. Browse all providers
EvoFlow-RNA is a non-coding RNA language model built on a masked discrete diffusion formulation, developed at Atom Bioworks with collaborators at Duke University and Rensselaer Polytechnic Institute, and first posted in February 2025. It targets a gap in RNA modeling: BERT-style encoders such as RiNALMo and RNA-FM produce excellent representations but were never designed to emit sequences, while autoregressive generators such as GenerRNA write RNA strictly left to right — a poor fit for a molecule whose function depends on base pairs between distant positions.
Masked discrete diffusion resolves that tension. The forward process progressively replaces tokens with a mask symbol; the reverse process iteratively predicts the masked positions, so the model keeps the bidirectional attention that makes encoders good at representation learning while gaining a real sampling procedure. Because any subset of positions can be held fixed and the rest infilled, conditional design falls out of the same machinery: given an aptamer's binding recognition site, the model redesigns everything around it.
Rather than training from scratch, EvoFlow-RNA initializes from published RiNALMo checkpoints and continues training under the diffusion objective, preserving the inherited representation quality while adding generative capability.
The architecture is a BERT-style encoder-only transformer extending RiNALMo-150M: 20 attention heads per block, SwiGLU activations, rotary positional embeddings, and FlashAttention-2. Tokenization is character-level over the four nucleotides with uracil normalized to thymine, plus IUPAC ambiguity codes, over sequences up to 1,022 tokens. Pretraining data is the RNAcentral corpus of more than 30 million non-coding RNA sequences, used without additional filtering. Models were trained at three scales — 33M, 150M, and 650M parameters — each initialized from the corresponding RiNALMo checkpoint and trained for 100K updates with batches of 320K tokens (33M and 150M) or 1M tokens (650M).
On the 13-task BEACON benchmark, the 150M model sets the best reported score on three tasks — contact map prediction (74.05 P@L), mean ribosome load (86.28 R²), and CRISPR on-target activity (41.48 Spearman) — and matches RiNALMo closely elsewhere, confirming that diffusion fine-tuning does not erode inherited representations. It also beats RNAGenesis on several tasks despite being nearly six times smaller. For unconditional generation at length 100, self-planning sampling yields mean pLDDT 69.41, minimum free energy -48.21 kcal/mol, entropy 1.89, and 59.8% GC — much closer to native sequences (48.26, -35.83, 1.96, 49.6%) on composition than RiNALMo-150M (59.01, -30.12, 1.29, 29.5%). Redesigned aptamers docked with HADDOCK 3.0 often scored better than wild type.
The design capability targets RNA aptamer engineering, where the conventional route is several rounds of SELEX selection from random libraries. Scaffolding around a known recognition site generates focused candidate libraries that retain binding chemistry while exploring different global folds — a route to affinity or stability gains without rediscovering the binding site. The unconditional mode supplies diverse ncRNA libraries with native-like biophysical profiles for diagnostics and biotechnology screens, and the same checkpoint doubles as an embedding source for downstream structure, function, and engineering predictors, including guide RNA activity models.
EvoFlow-RNA extends masked discrete diffusion, already established for protein sequence modeling, into RNA, and shows the conversion is close to free: initializing from an existing encoder and continuing under a diffusion objective preserves benchmark performance while adding generation and infilling. That recipe is straightforward to reuse on other RNA encoders. The authors state the limitations plainly. Trained solely on non-coding RNA, the model underperforms on mRNA-centric tasks, and its weak distance map prediction sits oddly against its strong secondary structure and contact map results. No generated sequence or scaffolded aptamer has been tested in the lab, so the design claims rest on in-silico folding and docking proxies. Only the 33M checkpoint has been released rather than the 150M model carrying the benchmark results. The licensing is also inconsistent in a way that matters for reuse: the repository's licence file is CC BY-NC-ND, which forbids sharing adaptations, while its README describes CC BY-NC and invites others to build on the work, and the Zenodo checkpoint deposit is CC BY. The preprint is non-commercial.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.