NucleicBERT

NucleicBERT is a 404-million-parameter masked language model for RNA sequences, developed at Forschungszentrum Jülich with the Karlsruhe Institute of Technology and released as a preprint in September 2025. It addresses a structural asymmetry between protein and RNA modeling: the leading RNA structure methods depend on multiple sequence alignments or on scarce experimental 3D labels, and both are thin for the non-coding transcriptome. NucleicBERT sidesteps both by pretraining on raw sequence alone and fine-tuning task-specific heads on top of the frozen-then-tuned backbone.

The model treats nucleotides as tokens and RNA sequences as sentences, following the BERT recipe. Pretraining on roughly 30 million non-coding RNA sequences yields representations that transfer to secondary structure prediction, nucleotide contact and distance maps, splice site classification, ribozyme fitness regression, and natural-versus-shuffled sequence discrimination — all from a single input sequence, with no alignment step at inference time.

What distinguishes NucleicBERT from contemporaries such as RiNALMo, RNA-FM, and RNABERT is the emphasis on interpretability. The authors pair benchmark results with saliency analysis, layer-wise attention profiling, PHATE embedding visualization, and a controlled shuffle-perturbation study, arguing that the model recovers RNA biology from sequence statistics rather than memorizing dataset artifacts.

#Key Features

• Alignment-free inference: Predictions require only a single RNA sequence, removing the MSA construction step that dominates runtime in coevolution-based methods and that fails outright for RNA families with shallow alignments.
• One backbone, five downstream tasks: Secondary structure, contact/distance maps, splice sites, fitness, and shuffle detection all attach as lightweight heads to the same pretrained encoder, and the released codebase ships configs for each.
• Interpretability as a first-class result: Saliency maps, normalized per-layer attention, and perturbation experiments show early layers acting as general RNA feature extractors while middle layers specialize by task; contact prediction concentrates attention in layers 0–15, while shuffle detection stays uniform.
• Emergent functional organization: PHATE projections of pretrained embeddings separate RNA classes — including similarly sized piRNAs and siRNAs — despite no functional labels ever entering pretraining.
• Character-level tokenization with span masking: A 25-token vocabulary covering the four nucleotides, sixteen ambiguity codes, and five special tokens, masked both at 15% of individual tokens and in contiguous 4–8 token spans to encourage motif-scale learning.

#Technical Details

The encoder stacks 32 transformer layers with an embedding dimension of 1,024, 32 attention heads per layer, learned positional encodings, and a maximum sequence length of 1,024 tokens. Pretraining data came from the MARS database: of its ~1.7 billion sequences, the ~30 million carrying an ncRNA annotation were extracted and split 80/20 into training and validation. Training ran for 300 epochs on 192 A100-40GB GPUs at roughly 80 minutes per epoch, plateauing at 83.1% masked-token accuracy on validation against 7.1% for a non-pretrained baseline.

On secondary structure, NucleicBERT reaches F1 0.872 on ArchiveII600 and 0.649 on the harder bpRNA-1m TS0 set, against 0.744 and 0.564 for RNA-FM and 0.875 and 0.622 for RNAErnie+. Contact map prediction at ≥24-nucleotide separation is where the margin is clearest: top-L/5 precision of 0.568 versus 0.508 for RiNALMo and 0.163 for RNA-FM, with a non-pretrained ablation at 0.283. Splice site accuracy averages 0.948 across zebrafish, fly, worm, and plant benchmarks, ahead of SpliceBERT at 0.943 and DNABERT at 0.925. On CPEB3 ribozyme self-cleavage fitness, the pretrained model attains Pearson r = 0.89 against 0.67 without pretraining.

#Applications

The practical draw is annotation and structural triage of non-coding RNA where no alignment is available. Genome annotation projects for newly sequenced or understudied species can call splice sites without species-specific training data; structural biologists can use predicted long-range contacts as restraints or as a fast filter before committing to 3D modeling or experiment; and RNA engineers working on ribozymes and aptamers can screen mutational libraries against the fitness head. The explainability tooling makes the model usable for hypothesis generation, since saliency maps identify which nucleotides drove a given prediction.

#Impact

NucleicBERT arrives as a preprint that has not yet been peer reviewed, and its benchmark gains are uneven: it is essentially tied with RNAErnie+ on ArchiveII600 secondary structure while pulling clearly ahead on tertiary contact prediction, which is the task the alignment-free framing most directly targets. Its more durable contribution may be methodological — a systematic explainability audit of an RNA language model, a genre of analysis that had been applied to protein transformers but not comprehensively to RNA. Release practice is unusually complete for the field: the source code is Apache-2.0 on GitHub with a tagged v1.0.0 archive, and the base checkpoint plus four fine-tuned heads are on Zenodo under the same license. The 1,024-token context window caps applicable transcript length, and processed derivative datasets are available only on request from the authors.