DGRNA

RNA transcripts are not short strings. Pre-mRNAs, long non-coding RNAs and viral genomes run to thousands of nucleotides, and the self-attention that RNA language models inherited from BERT costs quadratic time and memory in that length. The usual accommodation is to truncate the sequence — which discards exactly the long-range base pairing that gives an RNA its fold. DGRNA takes the other route and replaces attention as the backbone: twelve bidirectional Mamba2 blocks, state-space layers whose selective scan costs linear time in sequence length, with a single flash-attention layer at the end to recover token-to-token mixing at a fixed cost.

The bidirectional part matters for a masked language model. A state-space scan is causal by construction, so each block runs one Mamba2 pass left-to-right and a second over the reversed sequence, then fuses the two through a linear projection and a residual connection — giving each nucleotide a view of the bases on both sides of it, which a plain unidirectional Mamba stack cannot supply.

DGRNA was posted as a preprint in November 2024 by Ye Yuan of the Key Laboratory of Biopharmaceutical Preparation and Delivery at the Chinese Academy of Sciences, Qushuo Chen of Digitalgene.AI, and Xiaoyong Pan of Shanghai Jiao Tong University. It sits alongside a family of transformer-based RNA encoders — RNA-FM, Uni-RNA, ERNIE-RNA and RiNALMo — at 100 million parameters, roughly a sixth of RiNALMo's 650 million.

#Key Features

• Linear-time sequence backbone: The Mamba2 state-space dual formulation scales linearly with sequence length rather than quadratically, so the maximum 2,048-nucleotide context is affordable on commodity hardware.
• Bidirectional scanning: Each block pairs a forward and a reversed state-space pass, giving the masked-language objective the two-sided context a causal scan alone would miss.
• Averaged-embedding heads: Length-averaged 768-dimensional embeddings fed into a two-layer MLP are enough for translation-efficiency and RNA-binding-protein tasks; downstream tuning freezes or unfreezes layers as the task needs.
• Cross-species splice site transfer: Trained on Spliceator's multi-species G3PO+ split, the [CLS] representation transfers to independent zebrafish, fruit fly, worm and Arabidopsis test sets.
• Trained on four consumer GPUs: Pretraining ran on four RTX 3090 cards under fairseq, a modest budget for a model of this reach.

#Technical Details

Pretraining data came from the MARS database, which aggregates RNAcentral, MG-RAST, the Genome Warehouse, MGnify and NCBI's nucleotide database. From 1.7 billion collected sequences, MMseqs clustering left 1.2 billion non-redundant ones; sequences between 3 and 2,048 nucleotides were kept and roughly 100 million sampled at random. Thymine was mapped to uracil and the alphabet extended to 16 symbols — the IUPAC bases plus a gap character — with [CLS] and [EOS] on top. Training used BERT-style masking — 15% of tokens corrupted, 80% of those replaced by [MASK], 10% by a random base, 10% left intact — with an inverse-square-root schedule, a base learning rate of 1e-4 and a batch size of 16. The stack ends in a FlashAttention-2 layer with 32 heads, emitting an L×768 embedding matrix.

Across six downstream tasks and eight benchmark datasets, DGRNA is competitive with models several times its size. On 5' UTR mean-ribosome-load regression it reaches R² 0.93 on Random7600 and 0.87 on Human7600, against 0.93/0.86 for RiNALMo and 0.85/0.79 for RNA-FM. On the PrismNet RNA-binding-protein benchmark it is best on 14 of 17 proteins, with mean AUPRC 0.889 versus 0.865 for ERNIE-RNA and 0.824 for RNA-FM. Endogenous translation-efficiency prediction gives a Spearman correlation of 0.78 against RiNALMo's 0.74. On Spliceator-derived splice site prediction it leads SpliceBERT on every independent species set except Arabidopsis acceptors — 0.9711 versus 0.9568 average F1 on zebrafish, for instance. Non-coding RNA classification reaches F1 0.98 on the non-coding_s1 set, matching Uni-RNA.

#Applications

The natural use is as a frozen feature extractor for RNA regulatory questions where labelled data is scarce: ranking 5' UTR variants by predicted ribosome load before ordering constructs, scoring candidate microRNA–mRNA interactions, mapping RNA-binding-protein footprints, and annotating splice junctions in newly assembled genomes with no curated splice catalog. The 2,048-nucleotide window covers most mature transcripts and UTRs intact, which matters for mRNA design where truncation would cut the very elements under optimization.

#Impact

DGRNA is one of the clearer demonstrations that RNA language modelling does not require attention as its backbone — a 100-million-parameter state-space encoder matches or beats 650-million-parameter transformers across regulatory and interaction benchmarks, and does so with a cost curve that stays flat as sequences lengthen. The release is narrow, though: the MIT-licensed repository holds an inference package and an embedding-extraction script but no pretraining, fine-tuning or downstream task code, and the checkpoint is distributed through a Google Drive folder rather than a versioned host. The loader reuses RNA-FM's checkpoint-fetching path, so the weights must be placed in the Torch hub cache by hand, and inference requires a GPU with pinned versions of mamba-ssm, causal-conv1d and triton. The work remains a preprint and has not been peer reviewed.