Loading…
RNA foundation models that learn their own character-level tokenization instead of fixed nucleotide or k-mer vocabularies. Sizes run 8M to 650M.
No providers recorded yet. Browse all providers
Every RNA language model has to decide what counts as a token. Single nucleotides give four symbols and no notion of a motif; fixed k-mers or codons impose a vocabulary somebody chose in advance, so a stem-loop that happens to span an awkward number of bases gets split across boundaries that were never designed for it. ChaRNABERT's answer is to stop choosing. Its front end is a modified Gradient-Based Subsequence Tokenization (GBST) layer that pools every contiguous block of one to M nucleotides at every offset, scores each candidate with a small network, and mixes them under a softmax. The segmentation is therefore differentiable and is learned end to end with the encoder instead of being fixed before training starts.
ChaRNABERT — abbreviated CRB in the paper — was released in November 2024 by the AI department of Nostrum Biodiscovery together with the Institute for Research in Biomedicine (IRB Barcelona) and the Barcelona Supercomputing Center, and was presented again at the ICLR 2025 LMRL workshop. Pretraining ran on EuroHPC Joint Undertaking allocations.
Earlier RNA encoders such as RNA-FM, RNABERT, UNI-RNA and RiNALMo each commit to a hand-picked tokenization and then buy performance with parameters. The argument here is that the tokenizer is itself a strong inductive bias: the authors report that gains flatten out past roughly 30–50M parameters, so a small learnable-tokenizer model stays competitive with far larger fixed-vocabulary ones.
The encoder is a BERT-style bidirectional transformer, running from 6 layers at width 320 for the 8M variant to 33 layers at width 1280 for the 650M variant, with 20 attention heads throughout; training used BF16 with DeepSpeed and ZeRO. Fitting the Hoffmann scaling form to the observed losses gives optimal model size growing as C^0.2279 and optimal token count as C^0.7720, with training loss flattening beyond about 10^16 FLOPs.
Evaluation covers the 13-task BEACON benchmark plus two tasks the authors added. On BEACON, CRB-50M reaches R² 0.61 on distance map prediction and R² 0.43 on structural score imputation, CRB-8M reaches R² 0.83 on APA isoform prediction, and CRB-33M reaches R² 0.90 on mean ribosome loading; RiNALMo retains an edge on secondary structure (F1 0.72 versus 0.66). On a CLIP-derived RNA-binding-protein classification set, ChaRNABERT scores F1 0.833 ± 0.015 against RiNALMo's 0.831 ± 0.001, an LSTM's 0.719 and a CNN's 0.770. On aptamer-protein interactions curated from the UTexas Aptamer Database, it reaches F1 0.791 ± 0.018 against RiNALMo's 0.744.
The evaluation set is aimed squarely at RNA therapeutics work: ranking aptamers against a protein target, identifying which RNA-binding protein a genomic window belongs to, scoring CRISPR guide on- and off-target activity, predicting degradation for mRNA vaccine design, and estimating alternative polyadenylation site usage and ribosome loading. Because the smaller checkpoints hold up well, groups without large GPU allocations can fine-tune a usable RNA encoder on modest hardware.
The result worth carrying forward is methodological: a learnable tokenizer recovers much of what fixed-vocabulary RNA models spend parameters on, which reframes tokenization as a design choice rather than preprocessing. That claim currently rests on a single preprint, self-described as a first version and work in progress, with no independent reproduction. It is also hard to check directly — the paper states that weights and inference code for ChaRNABERT-8M will be provided for academic research use and the remaining models on request, but no public repository, model hub entry or weight release exists, so the benchmark numbers stand on the authors' report alone.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.