5' UTR-LM

Overview

The 5' UTR-LM is a semi-supervised transformer language model trained specifically on 5' untranslated region (UTR) sequences — the RNA segments that precede the protein-coding region of an mRNA transcript. These short but functionally rich sequences play a critical role in regulating translation initiation: the 5' UTR is read by the ribosome before it encounters the start codon, and its structural and sequence features directly determine how efficiently a transcript is translated into protein. Despite their importance, 5' UTRs have received far less modeling attention than protein-coding sequences, and prior predictive methods were largely task-specific rather than generalizable.

Published in Nature Machine Intelligence in April 2024, the model was developed by Yanyi Chu, Dan Yu, Mengdi Wang, and colleagues at Princeton University, Stanford University School of Medicine, RVAC Medicines, and Zipcode Bio. UTR-LM addresses the gap by learning transferable representations of 5' UTR biology through a combination of unsupervised masked nucleotide modeling and supervised auxiliary tasks — secondary structure prediction and minimum free energy (MFE) estimation — that inject biophysical knowledge directly into the pretraining objective.

The model outperforms task-specific baselines on multiple benchmarks covering translation efficiency, mean ribosome loading, and mRNA expression level prediction. It also demonstrates strong performance in zero-shot and few-shot settings, and its learned representations support de novo sequence design validated by wet-laboratory experiments.

Key Features

• Multi-task pretraining with biophysical supervision: Beyond standard masked language modeling, UTR-LM incorporates secondary structure and MFE prediction as auxiliary pretraining objectives, enabling the model to encode RNA folding information alongside sequence context without requiring separate structure prediction tools at inference time.
• Cross-species generalization: Pretraining data spans endogenous 5' UTRs from five vertebrate species (human, rat, mouse, chicken, and zebrafish), promoting representations that capture conserved regulatory signals rather than species-specific idiosyncrasies.
• Unified multi-task inference: A single pretrained model can be fine-tuned for translation efficiency prediction, mean ribosome loading estimation, expression level quantification, and internal ribosome entry site (IRES) detection — tasks that previously required separate specialized models.
• Interpretable attention patterns: The model's attention weights recover biologically known regulatory motifs, including Kozak consensus sequences and upstream ATG (uAUG) positions, providing mechanistic grounding for its predictions.
• Experimentally validated sequence design: UTR-LM can score and rank novel 5' UTR sequences for translation efficiency, enabling computational design campaigns confirmed by luciferase assays in cell culture.

Technical Details

UTR-LM uses a 6-layer transformer encoder with 16 multi-head self-attention heads and 128-dimensional nucleotide embeddings, paired with a two-layer feed-forward predictor block for downstream tasks. Layer normalization and residual connections are applied throughout. This compact architecture was deliberately sized to match the relatively short length of 5' UTRs (typically 50–500 nucleotides), trading capacity for efficiency.

Pretraining combines three data modalities. First, 214,349 endogenous 5' UTR sequences were drawn from Ensembl across five species. Second, approximately 280,000 synthetic 50-bp random sequences from eight published experimental libraries were included, providing broad sequence space coverage and measured translation activity labels for supervised augmentation. Third, three endogenous human cell-line datasets (HEK293, PC3, and muscle) contributed 41,446 unique sequences with empirical ribosome loading measurements. The pretraining objective jointly minimizes masked nucleotide prediction loss, secondary structure prediction cross-entropy, and MFE regression loss.

On held-out benchmarks, UTR-LM improves Spearman correlation for mean ribosome loading prediction by up to 5% over the prior best method (MTtrans), and by up to 8% for translation efficiency and mRNA expression level prediction compared to Cao-RF. For IRES detection, AUPR improves from 0.37 (IRESpy) to 0.52. Training was performed on four Tesla V100 or P100 GPUs.

Applications

UTR-LM is directly applicable to two major research domains. In mRNA therapeutics and vaccine development, optimized 5' UTRs can substantially increase the amount of protein produced per mRNA dose — a key parameter for mRNA vaccines and protein replacement therapies. The model's design workflow, in which a library of 211 computationally proposed 5' UTRs was evaluated by luciferase assay in C2C12 cells and top candidates produced 32.5% more protein than a well-established therapeutic benchmark sequence, demonstrates a practical design-test cycle that can accelerate UTR engineering. In basic research, fine-tuned UTR-LM variants can be used to annotate endogenous regulatory elements such as IRESs — structured RNA elements that allow cap-independent translation — in genomic datasets, aiding functional annotation of the human transcriptome.

Impact

UTR-LM demonstrates that the language model pretraining paradigm, which has proven transformative for protein sequences, can be productively extended to short functional RNA sequences with the addition of biophysics-informed auxiliary objectives. By consolidating multiple 5' UTR prediction tasks into a single transferable model and validating computational designs experimentally, the work establishes a template for data-efficient representation learning in the non-coding RNA space. It also highlights a relatively underexplored but practically important target for foundation models: the regulatory sequences that determine whether a transcript is translated, not merely transcribed. Limitations include the model's focus on 5' UTRs exclusively (3' UTRs and coding sequence context are not modeled), the relatively small architecture compared to protein language models, and the cell-line specificity of wet-lab validations that may not generalize across tissue types.