BiRNA-BERT

An RNA language model that reads one nucleotide per token can say something about every base — exactly what secondary structure, distance map and torsion angle prediction require — but its 1,024-token context then covers barely a kilobase. A 4,000-nucleotide long non-coding RNA gets truncated, and every distal contact past the cutoff is thrown away. Byte-pair encoding fixes the length problem by merging recurring motifs into single tokens, roughly a sixfold compression, but the resulting embeddings no longer correspond to individual bases. RNA models have generally had to commit to one tokenizer and accept the loss that comes with it.

BiRNA-BERT trains a single encoder on both views at once. During pretraining the same sequence is masked and reconstructed twice — once as byte-pair tokens, once as bare nucleotides — and the two masked-language-modeling losses are summed, so one set of weights carries both vocabularies. At inference an adaptive rule chooses between them by input length: nucleotide tokens when the sequence fits the available token budget, byte-pair tokens when it does not, which is why no input ever has to be truncated. Attention with Linear Biases (ALiBi) replaces learned positional embeddings, so the usable context can be extended beyond the pretraining length without any continued training.

The model was developed by Md Toki Tahmid, Haz Sameen Shahgir, Sazan Mahbub, Yue Dong and Md Shamsuzzoha Bayzid at the Bangladesh University of Engineering and Technology, with collaborators at UC Riverside and the University of Maryland, College Park. It was first posted as a preprint in July 2024 and published in Communications Biology in 2025. It sits alongside RNA-FM and RiNALMo as a general-purpose RNA encoder, and adapts the ALiBi-plus-BPE recipe DNABERT-2 introduced for DNA.

#Key Features

• Adaptive dual tokenization: A length threshold tied to available GPU memory decides per sequence whether to encode nucleotide by nucleotide or with byte-pair merges, keeping short inputs granular and long ones intact.
• Two embedding granularities from one checkpoint: The same weights produce per-nucleotide embeddings for residue-level structural heads and compressed sequence-level embeddings for classification, with no second model to maintain.
• Length extrapolation without retraining: ALiBi's distance-decaying attention bias generalizes past the pretraining window; raising alibi_starting_size in the config lifts the default 1,024-token cap at load time.
• A measured account of what compression costs: An information-theoretic analysis puts the per-character entropy of byte-pair tokens at 0.7514 of nucleotide tokens — about 25% less information in exchange for six times fewer tokens — and the downstream results track that prediction.

#Technical Details

BiRNA-BERT is a 117M-parameter BERT-style transformer encoder trained with the MosaicBERT framework and FlashAttention. Pretraining used 36 million non-coding RNA sequences from RNAcentral, totalling 26.42 billion tokens, for two epochs on eight consumer 24 GB GPUs at a learning rate of 2 × 10⁻⁴ with a 0.06 warmup ratio and 1,600 samples per batch. The byte-pair vocabulary holds 4,096 tokens averaging 6.08 nucleotides each; the nucleotide vocabulary holds four. On a species-classification benchmark built from RNAcentral sequences averaging roughly 2,500 nucleotides and reaching 10,000, it scores an F1 of 0.804 against 0.620 for RiNALMo and 0.532 for RNA-FM, the gap coming from truncation. It leads on five of six cross-species miRNA-lncRNA interaction splits, on all five RNA-binding proteins in 101-nucleotide RNA-protein interaction accuracy, and on eight of nine m6A site datasets spanning brain, kidney and liver in human, mouse and rat. On nucleotide-level structure it reaches F1 0.71 on the bpRNA TS0 secondary structure test set, R² 0.82 on 3D distance maps, and lower torsion angle error than the specialized SPOT-RNA-1D on the validation and TS1 sets.

#Applications

The checkpoint loads through AutoModelForMaskedLM with trust_remote_code=True and serves as a frozen embedder or a fine-tuning backbone. Frozen embeddings alone separate RNA structural families in unsupervised clustering, with the advantage widening on the longest sequences. Groups working on long non-coding RNAs, miRNA target interactions, RNA-binding protein specificity, m6A methylation and RNA structure can attach a lightweight head, choosing nucleotide mode for per-base outputs and byte-pair mode when whole transcripts must fit in memory.

#Impact

BiRNA-BERT's main contribution is showing that tokenization, not parameter count, is often the binding constraint in RNA language modeling: it matches or exceeds models roughly six times its size on structure tasks while pretraining on eight consumer GPUs rather than a cluster of A100s, putting RNA foundation model training within reach of ordinary academic hardware. The trade-offs are explicit. Nucleotide-level tasks must run in nucleotide mode, since byte-pair merges destroy the one-token-per-base correspondence those heads need. RiNALMo remains ahead on Rfam family clustering at 30 and 100 classes, a gap the authors attribute to its having pretrained on Rfam directly. And neither the code repository nor the released weights carry a license, leaving reuse terms unsettled despite the artifacts being public.