ProtRNA

ProtRNA is an RNA language model built by transferring a protein language model across modalities rather than pretraining on RNA from scratch. Protein language models have benefited from a corpus of hundreds of millions of deeply conserved sequences; RNA has neither the same volume nor the same degree of sequence conservation, and RNA language models have accordingly lagged their protein counterparts. ProtRNA asks whether the evolutionary and physicochemical regularities that ESM-2 has already absorbed from protein sequence can be repurposed for this lower-resource modality.

The model was developed by Ruoxi Zhang, Ben Ma, Gang Xu, and Jianpeng Ma at the Multiscale Research Institute of Complex Systems, Fudan University, with Shanghai AI Laboratory. It was first posted in September 2024, presented at the AI for Nucleic Acids workshop at ICLR 2025, and published in Cell Systems in September 2025.

The result is a compact demonstration that cross-modality initialisation works. ProtRNA matches or exceeds RNA-native baselines such as RNA-FM and RiNALMo across several unrelated downstream tasks while training only a small fraction of its weights on a fraction of the RNA data those models consumed — an argument that in low-resource biological modalities, where to start may matter as much as how much data you have.

#Key Features

• Cross-modality initialisation: The network begins from ESM-2 650M weights — 33 transformer blocks with a 1280-dimensional embedding, pretrained on UniRef50/D — and adapts them to nucleotide sequence instead of learning RNA grammar from random initialisation.
• Selective parameter updating: The first 29 transformer layers are frozen; only the RNA vocabulary embeddings and layers 30 through 33 are trained, which is what yields the 1/8 trainable-parameter figure relative to the reference RNA language model.
• Extended nucleotide vocabulary: Lowercase tokens a, c, u, g, and x are added alongside ESM-2's uppercase amino-acid tokens, with x covering modified and unknown nucleotides, so protein and RNA alphabets coexist in one embedding table.
• Task-general representations: A single pretrained checkpoint supports secondary structure prediction, protein-RNA interaction prediction, and mean ribosome loading regression through lightweight task heads rather than per-task backbone retraining.
• Released weights and data: Code is available under Apache 2.0, and pretrained weights plus the downstream task datasets and fine-tuned heads are archived on Zenodo under CC BY 4.0.

#Technical Details

Cross-modality transfer pretraining uses masked language modelling on 6 million non-coding RNA sequences. That corpus was assembled from the most recent RNAcentral release, which contained roughly 38 million sequences, and reduced to representative sequences by MMseqs2 clustering at 90% identity — about one-sixth the training volume of the reference baseline RNA language model. Freezing the lower 29 layers preserves ESM-2's general sequence-processing machinery while the upper layers and the new vocabulary embeddings absorb nucleotide-specific structure.

Evaluation spans three tasks with no shared objective. Secondary structure prediction is assessed on bpRNA-1m using RnaBench metrics, protein-RNA interaction on PrismNet-style targets covering 17 RNA-binding proteins in HeLa cells scored by AUROC, and mean ribosome loading on the benchmark used by RiNALMo. In the protein-RNA interaction comparison, ProtRNA embeddings improve over sequence-only input for most of the 17 proteins, while RNA-FM features degrade performance for many of them and RiNALMo features largely fail to deliver a comparable gain. The pretrained checkpoint loads with a single load_pretrained_model call and auto-downloads from Zenodo.

#Applications

ProtRNA is a general-purpose embedding model for RNA sequence, useful wherever a downstream RNA task has too little labelled data to support training a large model directly. Concrete cases include predicting secondary structure for uncharacterised non-coding RNAs, identifying where RNA-binding proteins engage a transcript, and estimating translational output from 5' UTR sequence for mRNA therapeutic design. Because the checkpoint is released and the fine-tuning heads are small, groups without large compute budgets can attach a task head to frozen features and get competitive results.

#Impact

The contribution is as much strategic as empirical: ProtRNA shows that a well-trained model in a data-rich biological modality can seed a model in a data-poor one, which reframes the assumption that each biomolecule class needs its own foundation model trained from scratch. That argument extends naturally to other sparse modalities. The practical limits are worth stating plainly — the frozen ESM-2 trunk fixes the architecture and context handling inherited from a protein model, the evaluation covers three tasks rather than a broad benchmark suite, and the gains over RNA-native models are framed in terms of parameter and data efficiency rather than a decisive accuracy margin.