mRNA2vec

A ribosome does not stop reading at the start codon. Translation initiation is decided in a window that straddles the AUG — the tail of the 5' untranslated region on one side, the first codons of the coding sequence on the other — yet mRNA language models are usually built for one side or the other. 5' UTR-LM sees only the untranslated region; CodonBERT sees only codons. Whichever half a model is given, the regulatory context on the far side of the start codon is absent from its input.

mRNA2vec closes that seam by concatenating the 5' UTR and the coding sequence into a single tokenized input and pretraining over the joined sequence, with the masking distribution deliberately concentrated on the junction region. Its second departure is the pretraining objective: rather than classifying masked tokens, it uses the data2vec teacher-student scheme, in which a student network reading the masked sequence regresses onto contextual representations produced by an exponential-moving-average teacher reading the unmasked one. The target is a whole-sequence representation rather than a token identity, which suits downstream tasks that consume an embedding of the entire region.

The model was developed by Honggen Zhang and June Zhang at the University of Hawaiʻi at Mānoa with Xiangrui Gao and Lipeng Lai at XtalPi, during an internship at the company. It was posted to arXiv in August 2024 and published at AAAI-25.

#Key Features

• Joint UTR-CDS input: The 5' UTR and coding sequence are tokenized separately and then concatenated, so the representation of a transcript carries the Kozak-context signal that spans the start codon rather than truncating at it.
• Location-aware probabilistic masking: Tokens inside a designated important region — tokens 15 through 45, covering the UTR tail and CDS head — are masked with elevated probability, so the objective spends its capacity where translation is regulated.
• Contextual teacher-student targets: The data2vec objective regresses student representations of a masked sequence onto averaged hidden states of the teacher's unmasked pass, richer supervision than masked-token prediction for tasks evaluated on whole sequences.
• Thermodynamic pretext tasks: Minimum free energy is learned as a ranking problem with an InfoNCE loss, and secondary structure as a token-wise 27-way classification over triplets of dot-bracket characters — a formulation that improves results where 5' UTR-LM had reported structure supervision to be unhelpful.
• Codon-granularity tokenizer: A 64-token vocabulary of nucleotide triplets matches the reading frame the ribosome uses, keeping sequences inside a 128-token context.

#Technical Details

The student is a T5 encoder with 4 attention heads, 4 hidden layers and a 256-dimensional token embedding — about 10M parameters in total, of which roughly 6M sit in the EMA teacher and do not receive gradients, leaving 3M trainable. Pretraining used 510k transcripts drawn from human, rat, mouse, chicken and zebrafish via the NCBI Datasets API, averaging 459 bp with the 5' UTR truncated to 102 nucleotides; minimum free energy and secondary structure labels came from RNAfold in the ViennaRNA package. The auxiliary losses are weighted at 0.01 and 0.001, and the second-to-last hidden state serves as the embedding, outperforming the final layer. The run took about three hours on four RTX 4090 GPUs.

On the three 5' UTR datasets from human embryonic kidney, PC3 prostate and muscle tissue, mRNA2vec reaches Spearman correlations of 0.68, 0.71 and 0.75 for translation efficiency and 0.69, 0.70 and 0.80 for expression level, against 0.60/0.63/0.66 and 0.65/0.55/0.61 for 5' UTR-LM and 0.62/0.70/0.75 and 0.58/0.60/0.78 for the 650M-parameter RiNALMo. On coding-sequence tasks it raises mRNA stability prediction on a 25k-sequence set from 0.34 to 0.53 over CodonBERT and matches it on E. coli mRFP protein production, a species absent from pretraining. RNA-FM and RNABERT were also evaluated as baselines.

#Applications

The model is used as a frozen or lightly fine-tuned encoder underneath a small regression head, scoring candidate sequences for mRNA vaccines and therapeutics before any are synthesized. Because one checkpoint covers both regions, a design team can rank 5' UTR variants for translation efficiency and expression, then rank synonymous codon choices in the coding sequence for stability and protein yield, without maintaining separate models for the two halves of the construct.

#Impact

mRNA2vec is an argument that architecture and objective design can substitute for scale on short regulatory sequences: 3M trainable parameters outperform a 650M-parameter general RNA language model on the UTR tasks, at a pretraining cost of a few GPU-hours. Its secondary-structure result also corrects a published negative finding, showing that a finer-grained classification target turns structure supervision from a liability into a consistent gain. The evaluation is entirely retrospective and computational — every reported number is a correlation against existing measurement datasets, with no wet-lab validation of designed sequences — and the released repository carries no license file, which limits how freely the code and checkpoint can be reused.