Loading…
RNA language model pretrained on 23 million non-coding sequences with motif-level masking, then steered by predicted RNA type at fine-tuning.
No providers recorded yet. Browse all providers
Masking a single nucleotide at random in an RNA sequence is usually easy to undo: local composition and flanking context narrow the answer to one or two bases. The elements that carry function — the four- to eight-nucleotide motifs that RNA-binding proteins and splicing factors recognise — are never demanded as a unit, because independent per-base masking almost always leaves most of a motif intact. RNAErnie corrupts sequences at three scales during pretraining: individual bases, contiguous subsequences of 4–8 nucleotides, and curated motifs harvested from experimentally characterised binding-site databases.
The model was built by the Big Data Lab at Baidu with collaborators at City University of Hong Kong, Shanghai Jiao Tong University, Nottingham Trent University and the Silesian University of Technology, and published in Nature Machine Intelligence in May 2024. It takes both its name and its backbone from ERNIE, Baidu's knowledge-integrating language model framework, whose whole-entity masking objective transfers naturally to RNA motifs.
A second idea closes the gap between a broad corpus and a narrow downstream task. RNAcentral labels each sequence with a coarse RNA type — miRNA, lncRNA, rRNA — and RNAErnie appends that label as a "stop word" token during pretraining. When a downstream sequence arrives unannotated, the model predicts its likely types, appends them, and feeds the augmented sequence to a task head, specialising the same weights post hoc rather than keeping a checkpoint per RNA class. That puts it alongside RNA-FM, RNABERT and RNA-MSM as a single reusable weight set. Despite the transposed name it is a separate model from ERNIE-RNA, which comes from a different group and injects base-pairing priors into attention instead.
RNAErnie is a 12-layer ERNIE transformer with a hidden dimension of 768, 12 attention heads, a 3,072-dimensional feed-forward block and a 512-nucleotide context; the paper reports roughly 105 million trainable parameters, while the PyTorch conversion redistributed through MultiMolecule counts 86.06 million. Pretraining used approximately 23 million non-coding sequences from RNAcentral under masked language modelling at a 15% masking rate, running about 2,580,000 steps with AdamW on ten Tesla V100 32GB GPUs over roughly 250 hours. On the nRC benchmark of 13 ncRNA classes it exceeds convolutional baselines such as ncRDense across accuracy, F1 and Matthews correlation coefficient; on the DeepMirTar miRNA–mRNA interaction set it reaches 0.9872 accuracy and 0.9976 AUC; and the stacked variant reaches 0.886 precision, 0.870 recall and 0.875 F1 for secondary structure on ArchiveII600. In June 2024 the first author re-pretrained the model from scratch in PyTorch at a 2,048-nucleotide context on about 31 million length-filtered RNAcentral sequences, released under the same name with a slightly different tokenisation.
The frozen backbone is the most common entry point: extracted embeddings drop into pipelines for non-coding RNA family assignment, miRNA target prediction and RNA-binding protein site prediction, replacing k-mer counts and hand-built sequence features. Groups working on RNA therapeutics can use the interaction head to triage candidate miRNA–mRNA pairs before reporter assays, and structural biologists can use the secondary structure head to propose base-pairing topologies for uncharacterised transcripts pulled from assemblies or screens where no curated annotation exists.
RNAErnie is one of the reference RNA language models later work benchmarks against, and downstream tools consume its frozen embeddings directly rather than treating it as a leaderboard entry. Its demonstration that curated motifs make a useful masking unit has carried into subsequent nucleotide models, and the type-token trick avoids training separate models per RNA class. The limitations are concrete: the original checkpoint drops sequences longer than 512 nucleotides and chunks longer transcripts, discarding long-range pairing information; RNA–protein interaction and binding-site identification are outside its scope; three-dimensional motifs such as loops and junctions are not modelled; and the stacking architecture multiplies inference cost by the number of candidate types. Distribution is fragmented too: the PaddlePaddle weights and per-task checkpoints live in Google Drive folders rather than versioned artifact hosting, while the PyTorch lineage sits on Hugging Face.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.