mRNABERT

A messenger RNA is one molecule, but the models built to design it have not been. A 5' UTR model such as 5' UTR-LM reads single nucleotides, which preserves the structural signal in untranslated regions but exhausts a transformer's context long before a full transcript is consumed. A coding-sequence model such as CaLM reads codons, which triples the effective context and exposes synonymous usage, but applied to a UTR it slices a non-triplet region into meaningless triplets. Designing a complete construct therefore meant running one model per region and hoping the pieces composed — even though the 5' UTR, CDS and 3' UTR determine translation efficiency and stability jointly.

mRNABERT, from Tingjun Hou's group at Zhejiang University, resolves this by applying both tokenizers inside a single sequence: individual nucleotides for the untranslated regions, codons for the coding sequence. One vocabulary of 74 tokens carries both granularities, so a single encoder sees the whole transcript at codon-compressed length without garbling its flanks. A second training stage then aligns the model's codon embeddings with protein language model embeddings of the same sequences translated, which is what lets an mRNA model compete on protein-level property prediction.

An unrelated encoder also called mRNAbert — pretrained by the BioCiphers Lab at the University of Pennsylvania as the backbone of G4mer — is a different model by different authors on a different task, and results reported for one do not transfer to the other.

#Key Features

• Dual tokenization within one transcript: UTRs are tokenized per nucleotide and the CDS per codon, so local structural detail and codon-level semantics coexist in the same forward pass rather than requiring two specialized models.
• Cross-modality contrastive alignment: a CLIP-style objective pulls each coding sequence toward the embedding of its translated protein, produced by a frozen ProtT5-XL-UniRef50. Synonymous codons cluster by amino acid afterwards, and chemically similar amino acids group together (adjusted Rand index rising from 0.166 to 0.498).
• Length extrapolation via ALiBi: linear attention biases replace learned positional embeddings, so the model accepts sequences several times longer than the 1022 tokens it was trained on.
• One backbone across mRNA regions: the same checkpoint is fine-tuned for ribosome load, CDS expression and stability, RBP binding, m6A sites, splice sites, alternative polyadenylation and full-length transcript properties.

#Technical Details

mRNABERT is a 12-layer bidirectional transformer with 768-dimensional hidden states, configured like DNABERT-2 with ALiBi positions and FlashAttention. Its vocabulary holds 64 codons, five nucleotides and five special tokens. Pretraining aggregated roughly 36 million complete-CDS mRNA sequences from NCBI's nt database (over 70% of the total), MG-RAST, GWH and MGnify, resolving CDS boundaries with NCBI ORFfinder where annotation was absent, then filtered to about 18 million unique sequences capped at 1022 tokens. Masked language modeling ran at a 15% masking ratio for 660,000 gradient steps over 10 epochs on NVIDIA A6000 GPUs; the contrastive stage used 500,000 coding sequences projected to 256 dimensions alongside their translations.

Across six CDS tasks it matched or led every baseline, scoring Spearman 0.89 on mRFP and fungal expression, 0.56 on mRNA stability and 0.89 on SARS-CoV-2 vaccine degradation. On eight 5' UTR ribosome-load libraries it reached Spearman 0.962 and 0.924 on the two largest, leading four of eight tasks. Over 22 RNA-binding proteins it averaged 0.786 accuracy and 0.501 MCC. Contrastive learning raised protein melting-point prediction from R² 0.60 to 0.77, above ESM-2 and ProtTrans variants, and translation-efficiency prediction on transcripts averaging about 4000 nt reached mean R² 0.66 at a 3066-token window.

#Applications

The model is aimed at mRNA therapeutic and vaccine design, where UTR and codon choices are ranked computationally before any construct is synthesized. It supports 5' UTR selection by predicted ribosome load, codon optimization for expression and stability, RBP and m6A site annotation in 3' UTRs, splice site and polyadenylation analysis, and whole-transcript scoring for translation efficiency and degradation. Its embeddings also serve protein-side work — melting point, solubility and transcript abundance — for groups that hold coding sequences rather than curated protein datasets.

#Impact

mRNABERT's argument is that curation and tokenization buy more than scale: an 18-million-sequence corpus and a hybrid vocabulary let a 12-layer encoder match or beat far larger protein language models on codon-level tasks. Both the pretraining corpus and the downstream benchmark datasets are deposited publicly, so the comparison is reproducible rather than asserted. The limits are real ones. Region-specialized models still edge it in places — 3UTRBERT on m6A sites and ERNIE-RNA on splice sites — and the full-length evaluation rests on 233 reporter constructs, a small set for the task it is meant to represent. Every result is computational; no designed sequence in the paper was synthesized and assayed.