lncRNA-BERT

Every RNA sequencing experiment that assembles novel transcripts ends with the same question: is this a messenger RNA or a long non-coding RNA? The distinction matters because lncRNAs turn out to carry substantial regulatory function, and the established answer — coding potential classifiers such as CPAT, LncFinder, and RNAsamba — rests on hand-designed features like open reading frame length, Fickett score, and hexamer bias. Those features encode a prior about what coding sequence looks like, which is exactly what breaks down on the ambiguous transcripts that most need classifying.

lncRNA-BERT replaces the feature engineering with pretraining. Built at the Department of Biomedical Data Sciences at Leiden University Medical Center with Leiden's computer science institute, it is a BERT-style nucleotide language model trained by masked language modelling on human mRNA and lncRNA from GENCODE, RefSeq, and NONCODE, then fine-tuned for classification. Its notable empirical result is that fine-tuning is not strictly required: the pretrained model alone separates coding from long non-coding transcripts, which is direct evidence that coding potential is a sequence-intrinsic property rather than an artefact of the labelling scheme.

The second contribution is architectural. Transcripts routinely run to several thousand nucleotides, well past what k-mer tokenisation or byte-pair encoding can fit into a standard context window, so the authors introduced Convolutional Sequence Encoding to compress the input before the transformer ever sees it.

#Key Features

• Convolutional Sequence Encoding: A single 1D convolution with stride equal to its kernel width slides over the one-hot nucleotide input, turning each non-overlapping window into a continuous embedding — the vision transformer's patching idea applied to nucleotides, with no discrete vocabulary at all.
• Long transcripts in one pass: At a kernel width of 9 and a 768-position context, a single forward pass covers 6,912 nucleotides, roughly triple what 3-mer tokenisation reaches at the same context length.
• Unsupervised separation of coding potential: Coding and non-coding transcripts occupy distinguishable regions of the pretrained embedding space before any labelled fine-tuning.
• Curated human pretraining corpus: Pretraining on GENCODE, RefSeq, and NONCODE outperformed the same configurations pretrained on the more commonly used RNAcentral collection.
• Shipped as a working library: lncRNA-Py bundles the model with re-implementations of the coding-potential features used by more than forty published classifiers, so learned and classical approaches can be compared on equal footing.

#Technical Details

The released model follows a BERT-medium configuration — 12 transformer blocks at a model dimension of 768, 12 attention heads, and a feed-forward width of 3,072 — chosen deliberately to match DNABERT-2, BiRNA-BERT, and GENA-LM for fair comparison. The convolutional encoder learns 768 kernels of width 9 with ReLU activation and no linear projection, a configuration arrived at by ablation: adding kernels beyond the model dimension did not help, and masking spans of more than one nucleotide during pretraining did not either. Training data was assembled by pooling GENCODE v46 protein-coding and lncRNA transcripts with NONCODE lncRNAs and human RefSeq, discarding anything under 100 bases, and holding out disjoint GENCODE splits for fine-tuning, validation, and test. Pretraining ran for seven days on a single A100 partition.

Evaluation used macro-averaged F1 against three classical predictors (CPAT, LncFinder, PredLnc-GFStack) and three deep learning ones (LncADeep, mRNN, RNAsamba) across three test sets. On the CPAT test set every method lands between 0.950 and 0.970, with lncRNA-BERT at 0.963 — the benchmark is saturated. The separation appears on RNAChallenge, a cross-species set built from the transcripts that existing classifiers get wrong, where lncRNA-BERT outperforms all of them.

#Applications

The immediate use is annotating transcripts from RNA-seq assemblies, where a large fraction of novel entries have no ortholog and no functional evidence to lean on. Because RNAChallenge is cross-species, the model is also usable outside human data despite being pretrained only on human transcripts, which matters for non-model organisms with sparse annotation. Beyond classification, the Convolutional Sequence Encoding scheme is the transferable part: any nucleotide language model constrained by context length — genomic, transcriptomic, or otherwise — can adopt a stride-equals-kernel convolutional front end in place of tokenisation.

#Impact

lncRNA-BERT's clearest scientific claim is the unsupervised one: that a model trained only to fill in masked nucleotides organises transcripts by coding potential without being told the labels exist. Its practical scope is narrower than that suggests — one task, one species in pretraining, and a benchmark suite where two of three test sets no longer discriminate between methods. Weights for both the convolutional and 3-mer variants are on Hugging Face in pretrained and fine-tuned form, and the package is MIT licensed. The preprint appeared in January 2025 and the peer-reviewed version in Connection Science.