Loading…
RNA secondary structure prediction from a single sequence, without MSAs. Axial-attention transformer reaching F1 0.764 on the TS-PDB benchmark.
No providers recorded yet. Browse all providers
Transfer RNAs from distant organisms fold into the same clover leaf while sharing very little sequence identity. Split a training set on sequence similarity alone and near-identical structures land on both sides of the line, so a model scores well on a held-out set it has already seen. That is why deep learning stayed contentious in RNA secondary structure prediction: reported gains were hard to separate from leakage.
RNAformer, built by the Department of Computer Science at the University of Freiburg, pairs a deliberately lean architecture with a data pipeline designed to expose that leakage. It predicts base pairing from a single RNA sequence — no multiple sequence alignment, no ensemble, no thermodynamic post-processing — by embedding the sequence twice, once row-wise and once column-wise, and broadcasting the two embeddings into a two-dimensional latent matrix shaped like the adjacency matrix it has to produce. A stack of axial-attention blocks refines that matrix, and because the latent space is already pairwise, a linear layer and a sigmoid read out the pair probability matrix.
The pipeline separates training from test data on structure similarity as well as sequence identity, using covariance models to catch homologs that identity cutoffs miss. Trained under those controls, RNAformer leads published single-sequence methods on unseen families, and on secondary structure extracted from tertiary predictions it scores above AlphaFold 3.
Each RNAformer block combines row-wise and column-wise axial attention with a convolutional transition network using SiLU activations, wrapped in residual connections and pre-layer norm; rotary position embeddings replace absolute positional encoding. Because adjacency matrices are overwhelmingly zero, training uses a masked binary cross-entropy loss keeping all true pairings and only about 40% of the zero entries — 80% during fine-tuning. Models were trained at 2M, 8M and 32M parameters with FlashAttention, and sequences capped at 500 nucleotides.
Pretraining on synthetic data — 410,408 samples from 3,796 Rfam 14.9 families, folded with RNAfold — showed the architecture can absorb a biophysical folding model, the 32M variant recovering RNAfold's structures at F1 0.967 and reproducing 83.5% of them exactly. On experimental data, a pool of 107,098 sequences from bpRNA-1m, ArchiveII, RNAStrAlign, RNA-Strand and the PDB was filtered to a 66,242-sample pretraining set, then fine-tuned on FT-Non-Homolog: 3,432 PDB structures with no sequence or structure homology to either test set. That model reaches F1 0.764 on TS-PDB (125 samples pooled from TS1–TS3) and 0.679 on TS-Hard, ahead of the MSA-dependent SPOT-RNA2 (0.754 and 0.666) and RNA-FM (0.729 and 0.665). Fine-tuned instead on FT-Homolog — 4,244 PDB entries cut off at September 2021 to match AlphaFold 3's training window — it scores 0.855 and 0.845 against AlphaFold 3's 0.817 and 0.688.
RNAformer suits any context requiring fast, reference-free RNA secondary structure prediction. Structural biologists and RNA biochemists can annotate newly sequenced non-coding RNAs, riboswitches, or viral genomes where MSA construction is impractical. Medicinal chemists working on RNA-targeted therapeutics can locate candidate drug-binding pockets in undercharacterized RNAs, and synthetic biologists designing aptamers, ribozymes, or RNA scaffolds can check whether a sequence adopts the intended fold. Inference is a single forward pass, making the model practical for transcriptome-scale sequence sets or mutational series.
RNAformer's clearest contribution is methodological: splitting on structure as well as sequence, and reporting under those stricter controls, gives the field a cleaner basis for comparison than benchmarks criticized for homology leakage. The architectural argument matters too — one 32M-parameter model with no MSA, ensemble or post-processing outperforming systems that use all three suggests earlier pipelines' complexity was compensating for data problems rather than modeling ones. The limits are real: the two-dimensional latent representation carries a large memory footprint, training was capped at 500 nucleotides, and predictions cover base pairing rather than three-dimensional coordinates. The published checkpoints come from the earlier synthetic and family-split experiments; the homology-aware fine-tuned models behind the headline numbers have not been posted.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.