mRNAbert

Experimental assays that label RNA behaviour return thousands of sequences, not millions. The reprocessed rG4-seq dataset behind G-quadruplex prediction, for instance, contributes about five thousand labelled windows. A transformer fit directly on a set that size has to learn two things at once from the same handful of examples: what human mRNA sequence looks like in general, and what the particular assay measures. mRNAbert splits the two apart. It learns the first on its own, by masked language modeling over the entire human transcriptome, and leaves only the second to the labelled data.

The encoder was pretrained by the BioCiphers group at the University of Pennsylvania and introduced as the general half of the G4mer work — first on bioRxiv in October 2024, then in Nature Communications in November 2025. G4mer is this checkpoint fine-tuned for RNA G-quadruplex formation, subtype classification and variant effect scoring, and it is what demonstrates that the pretrained representation transfers: the same fine-tuned weights hold up on 795 held-out G4RNA sequences spanning 24 experimental protocols and lengths from 14 to 1368 nucleotides, none of which the fine-tuning set contained.

The pretraining corpus is deliberately narrow. It is GENCODE GRCh38 mature transcript sequences and nothing else, so mRNAbert is a model of the human transcriptome rather than a pan-species RNA language model, and its representations carry human splicing, codon and untranslated-region statistics specifically. A separately developed model published in 2025 by a group at Zhejiang University uses the same name for mRNA sequence design; the two are unrelated in authorship, task and artifacts.

#Key Features

• Transcriptome-wide self-supervision: masked-token prediction over every mature human transcript, deduplicated before training, so no assay labels or task annotations are needed to learn the representation.
• Overlapping 6-mer tokenization: sequences are tokenized into overlapping six-nucleotide k-mers rather than single bases, a scheme chosen for the size and character of mRNA data, which keeps local sequence dependencies inside one continuous token stream.
• Long transcripts are split, not truncated: a transcript exceeding the 510-token input is divided into separate non-overlapping segments, on the reasoning that the overlapping k-mers already capture motifs across the boundary — full-length mRNAs are therefore represented rather than clipped at the first 512 tokens.
• Compact enough to fine-tune locally: six transformer layers instead of BERT-base's twelve puts the released checkpoint near 47M parameters, small enough to fine-tune and run inference on a single GPU.
• Demonstrated downstream transfer: fine-tuning the encoder on roughly five thousand rG4 windows produces a predictor that generalizes across assay chemistries and two orders of magnitude of sequence length.

#Technical Details

mRNAbert is a modified BERT encoder: six transformer layers with six attention heads each, a reduction from the default architecture chosen to suit the size of the mRNA corpus. Input sequences are converted to overlapping 6-mer tokens capped at 510 per input, leaving room for the [CLS] and [SEP] tokens within a 512-token context. Pretraining used mixed-precision arithmetic on four NVIDIA Tesla P100 GPUs over three weeks: 200,000 steps at a learning rate of 4e-4 with an effective batch size of 250, using gradient accumulation and learning rate scheduling. The released checkpoint is a BertForMaskedLM model of roughly 47M parameters, distributed through HuggingFace in PyTorch format with its 6-mer vocabulary and tokenizer configuration.

#Applications

The model is a starting point for supervised tasks on human mRNA sequence where labelled examples are scarce — structural propensity, regulatory element detection, variant effect scoring in untranslated regions. The worked example is rG4 prediction, where the fine-tuned descendant scores transcriptome-wide G-quadruplex formation and was applied unchanged to gnomAD, ClinVar and Penn Medicine BioBank variants. Groups with their own RNA assay data can attach a classification or regression head with a few lines of the standard HuggingFace fine-tuning loop, which is what the model card documents.

#Impact

mRNAbert is a useful reference point for how much pretraining a domain-specific RNA encoder actually requires: six layers and three weeks on four older-generation GPUs was enough to support a downstream predictor that beats convolutional baselines fit on the same labels. Its published evaluation, however, is entirely downstream — the paper reports G4mer's benchmark results, not an intrinsic language-modeling assessment of the encoder. Pretraining code is not part of the released Bitbucket repository, which covers fine-tuning, prediction and analysis, so reproducing the encoder from scratch is not currently possible. Weights are distributed under a University of Pennsylvania academic license permitting internal, non-commercial use by individuals at academic or not-for-profit institutions, and require accepting that license before download; commercial use requires contacting the corresponding author.