RiNALMo

Overview

RiNALMo (RiboNucleic Acid Language Model) is a general-purpose RNA foundation model developed by the Laboratory for Bioinformatics and Computational Biology (LBCB) at the University of Zagreb. With 650 million parameters pre-trained on 36 million non-coding RNA sequences, it stands as one of the largest RNA language models trained to date. The model was introduced in a February 2024 preprint and subsequently published in Nature Communications in 2025, establishing it as a peer-reviewed benchmark in the RNA foundation model landscape.

The central motivation behind RiNALMo was to address a persistent limitation in RNA secondary structure prediction: the inability of existing deep learning methods to generalize beyond the RNA families seen during training. Prior approaches, even those leveraging sequence co-evolution or deep learning, would often fail when presented with RNA families absent from their training sets. RiNALMo was designed to overcome this by learning broad, transferable representations of RNA sequence grammar from a large and diverse corpus of non-coding RNAs, following a strategy inspired by protein language models such as ESM-2.

The model was pre-trained entirely without structural supervision, using masked language modeling on raw RNA sequences. This forces the network to capture the implicit structural and functional information encoded in sequence patterns — an approach that, once validated at scale, proved capable of driving state-of-the-art performance across a wide range of downstream prediction tasks with only lightweight fine-tuning.

Key Features

• Strong inter-family generalization: RiNALMo generalizes to unseen RNA families in secondary structure prediction, addressing a critical failure mode of prior deep learning methods. Across nine benchmarked RNA families, it achieves a mean F1 score of 0.72, with particularly strong performance on tRNA (F1 0.93) and 5S rRNA (F1 0.88).
• Multi-task fine-tuning capability: A single pre-trained model can be fine-tuned for diverse tasks including secondary structure prediction, splice-site detection, mean ribosome loading, translation efficiency, expression level prediction, and non-coding RNA family classification.
• Modern transformer architecture: Incorporates rotary positional embeddings (RoPE), SwiGLU activation functions, and FlashAttention-2 for computational efficiency, reflecting best practices from recent large language model development.
• Model family at multiple scales: Three model variants are provided — Giga (650M parameters), Mega (150M), and Micro (35M) — allowing researchers to match compute requirements to their task and hardware constraints.
• Permissive licensing: Code is released under Apache 2.0 and model weights under Creative Commons Attribution 4.0 (CC BY 4.0), enabling broad use in both academic and commercial contexts.

Technical Details

RiNALMo's architecture is a BERT-style transformer encoder comprising 33 transformer blocks. Input RNA sequences are tokenized and projected into a 1280-dimensional embedding space before being processed by the attention stack. Each transformer block combines multi-head self-attention with a feed-forward network, using RoPE for positional encoding in place of absolute positional embeddings. The use of SwiGLU activations and FlashAttention-2 improves training stability and GPU utilization at scale.

Pre-training used masked language modeling (MLM) with 15% of input tokens masked per sequence. Training data consisted of 36 million non-coding RNA sequences curated from RNAcentral, Rfam, NCBI Nucleotide, and Ensembl, providing broad coverage of RNA families and species. On secondary structure benchmarks, RiNALMo achieves an F1 of 0.75 on the intra-family TS0 dataset, outperforming SPOT-RNA (0.64), UFold (0.66), MXfold2 (0.60), and RNA-FM (0.68). On multi-species splice-site prediction, it achieves average F1 scores of 97.70 (fish), 96.25 (plant), 96.11 (fly), and 95.63 (worm), exceeding SpliceBERT across all tested species. For mean ribosome loading prediction, RiNALMo reaches R² values of 0.93 and 0.86 on the Random7600 and Human7600 datasets, respectively.

Applications

RiNALMo is well suited for any research context where high-quality RNA sequence representations are needed. Structural biologists can use fine-tuned checkpoints to predict RNA secondary structures, particularly for novel RNA families where homology-based methods or alignment-dependent tools underperform. Molecular biologists studying post-transcriptional regulation can leverage the model for splice-site prediction and translational efficiency estimation. Functional genomics researchers can apply it to ncRNA family classification or expression level prediction. The availability of the Micro and Mega variants makes the model accessible for research groups without access to high-end GPU clusters, while the Giga variant provides maximum representational capacity for demanding tasks.

Impact

RiNALMo's publication in Nature Communications and its strong benchmark results helped establish that RNA language models trained at sufficient scale can achieve meaningful generalization — a claim that had remained uncertain given the greater structural complexity of RNA relative to proteins. The model directly addresses one of the core criticisms of earlier RNA deep learning methods and positions the field to benefit from continued scaling. Its multi-scale model family and permissive licensing have encouraged adoption across academic groups, and third-party integrations such as the MultiMolecule HuggingFace repository have extended its accessibility further. A notable current limitation is that RiNALMo, like most RNA language models, predicts secondary structure rather than tertiary or 3D structure, and does not model RNA-protein interactions or dynamic conformational states — areas where dedicated tools such as RhoFold+ or trRosettaRNA remain complementary.