DLRNA-BERTa

Small molecules that bind structured RNA are an expanding therapeutic modality, but measured RNA–ligand affinities remain orders of magnitude scarcer than protein–ligand data, which has kept computational RNA–drug screening tied to small hand-engineered feature sets. DLRNA-BERTa, developed at the Institute for Molecular Medicine Finland (FIMM) and HiLIFE at the University of Helsinki, addresses that gap by pairing two pretrained language models — one for RNA, one for chemistry — and fine-tuning them jointly to regress binding affinity directly from an RNA FASTA sequence and a compound SMILES string.

The framework's central contribution is RNA-BERTa, a RoBERTa-style RNA language model the authors pretrained from scratch on 9,757,119 RNA sequences, joining a growing set of RNA foundation models such as RNA-FM and RNABERT. RNA-BERTa's target embeddings are fused with molecular embeddings from ChemBERTa-2 through a single-head cross-attention interaction layer that outputs a continuous pKd value. Rather than one monolithic predictor, the release comprises seven fine-tuned models: six specialized for aptamers, repeat expansions, ribosomal RNAs, riboswitches, miRNAs, and viral RNAs, plus a general model for targets whose class is unknown.

DLRNA-BERTa was posted to bioRxiv in September 2025 and has not yet been peer reviewed. The RNA-BERTa backbone is distributed as a standalone Hugging Face checkpoint under CC BY-NC 4.0, and the fine-tuned interaction models, training code, and datasets are released through GitHub and a hosted Gradio application.

#Key Features

• Dedicated RNA backbone: RNA-BERTa is pretrained specifically for this task rather than borrowed, and is usable on its own as a general-purpose RNA sequence encoder for embeddings or downstream fine-tuning.
• Cross-attention fusion: A single-head cross-attention layer over 384-dimensional projections lets each drug token attend to target tokens, replacing the concatenation of independently pooled encoder outputs used by earlier RNA–drug regressors.
• Class-specific checkpoints: Six RNA-class models plus a general model let users match the predictor to the target biology, with the general model as a fallback for uncharacterized RNAs.
• Interpretable predictions: Two pooling variants ship with the release — mean pooling and an interpretable target-token pooling — exposing cross-attention heatmaps and per-token contribution plots that indicate which nucleotides drive a predicted affinity.
• Multiple access routes: A public Gradio Space accepts batches of up to 2,000 SMILES against one target, with a Colab notebook and a Dockerfile for local or GPU-backed runs.

#Technical Details

RNA-BERTa is a 55.9M-parameter RoBERTa encoder with a 512-token context window and 512-dimensional embeddings, trained with masked language modeling on roughly 1.07 billion tokens drawn from RNACentral, NCBI Virus, and SILVA, filtered to sequences of at most 2,000 nucleotides. Its parameter budget follows the Chinchilla-style 1:20 parameter-to-token ratio, and learning rate and warm-up were tuned by 32 Optuna trials on a smaller proxy model using maximal update parametrization (μP) for hyperparameter transfer, cutting tuning cost about 4.5-fold on four V100 GPUs. The drug encoder is ChemBERTa-77M-MTR. Fine-tuning used 1,439 RNA–drug pairs spanning 759 unique compounds and 294 unique RNA sequences with measured dissociation constants. Reported Pearson correlations between predicted and measured pKd range from 0.92 for the general model to 0.98 for miRNAs. On four independent ROBIN classification datasets totaling 5,534 pairs, AUROC falls to 0.57–0.60, with recall above 0.94 but specificity near zero — the models flag almost everything as active on those external sets. The authors report that DLRNA-BERTa outperforms prior RNA–drug affinity regressors, while noting that DeepRNA-DTI, a classification-based method trained on more data, scores higher on the ROBIN comparison.

#Applications

The intended use is early-stage virtual screening for RNA-targeted drug discovery: ranking compound libraries against a structured RNA of interest before committing to biophysical assays, and triaging approved drugs for RNA-binding activity. As a demonstration, the authors screened 3,492 approved drugs from ChEMBL against 294 RNA targets and flagged 2,859 compounds with predicted pKd of at least 6, highlighting bleomycin as a case with independent literature evidence of RNA binding. Because inference needs only a sequence and a SMILES string, the models are usable by medicinal chemists and RNA biologists without structural data or docking setup.

#Impact

DLRNA-BERTa contributes one of the few openly released RNA language models trained explicitly for ligand-binding tasks, and RNA-BERTa is reusable independently of the affinity head. The practical ceiling is the fine-tuning corpus: 1,439 measured pairs is small for a transformer pipeline, and the gap between in-distribution Pearson correlations above 0.9 and near-chance external AUROC shows the models are better at ranking within familiar target classes than at separating binders from non-binders on unseen RNAs. The public availability of code, weights, datasets, and a hosted application nonetheless makes it a usable baseline for a field where open RNA–small molecule predictors are scarce. Licensing is the main constraint on reuse: the RNA-BERTa weights carry a non-commercial license, and the code repository ships no license file at all.