RNA-xLSTM

RNA-xLSTM is a 33.7-million-parameter RNA language model that replaces the transformer backbone used by essentially every current RNA foundation model with xLSTM, a modernized long short-term memory architecture built around exponential gating and a matrix-valued memory cell. It was developed at the Laboratory for Bioinformatics and Computational Biology at the University of Zagreb — the group behind RiNALMo — in collaboration with the Genome Institute of Singapore, and released as a bioRxiv preprint in July 2025.

RNA sequence data is scarcer and less annotated than protein data, which is why self-supervised pretraining has become the dominant strategy for RNA representation learning, in models such as RNA-FM, Uni-RNA, and RiNALMo — all attention-based. Recurrent and state-space alternatives such as Mamba, RWKV, and xLSTM promise linear-time scaling and better handling of long sequences, and the Bio-xLSTM study extended xLSTM to DNA, protein, and chemical sequences. RNA was the gap, and RNA-xLSTM fills it as a controlled architectural comparison rather than a bid for state-of-the-art accuracy.

The result is a candid mixed one. Matched almost exactly in size to RiNALMo-33M and trained through RiNALMo's own data and fine-tuning pipeline, RNA-xLSTM outperformed the roughly three-times-larger RNA-FM on family-held-out secondary structure prediction while trailing RiNALMo, and underperformed both baselines on splice-site prediction. The authors conclude that xLSTM's advantages over transformers for RNA remain unclear.

#Key Features

• Recurrent backbone in place of self-attention: Fifteen stacked mLSTM blocks with exponential gating and a matrix memory replace the attention stack, avoiding quadratic scaling in sequence length.
• Bidirectionality by block-wise sequence flipping: Because mLSTM processes sequences directionally, the input is reversed after each block — the bidirectional scheme introduced in Bio-xLSTM — rather than relying on bidirectional attention.
• Parameter-matched, pipeline-matched comparison: 33.7M parameters against RiNALMo's 33.5M, with the same tokenizer, masking scheme, prediction heads, and layer-unfreezing schedules, isolating the backbone as the experimental variable.
• Generalization measured across RNA families: ArchiveII evaluation uses leave-one-family-out training, the harder protocol that exposes the family-memorization failure mode of intra-family benchmarks such as bpRNA TS0.
• Open code and weights, partial reproducibility: The Apache-2.0 repository includes fine-tuning scripts and automatic dataset download, with pretrained weights distributed separately; the pretraining script and data-preparation code are not released.

#Technical Details

The architecture adapts the DNA-xLSTM-4M configuration from Bio-xLSTM, scaled up in depth and width: 15 mLSTM blocks, an embedding dimension of 600, rotary positional embeddings, and a 2:1 up-projection ratio. Only mLSTM blocks are used, since sLSTM blocks are not parallelizable during training. Pretraining data came from RNAcentral, filtered to 16-8,192 nucleotides, deduplicated with seqkit and clustered with MMseqs2 linclust at 60% identity and 0.8 coverage so each batch samples distinct clusters. The tokenizer follows RiNALMo, covering the four bases, IUPAC degenerate codes, a gap symbol, and special tokens. Training used BERT-style masked language modeling on 15% of tokens, sequences truncated to 1,024 tokens, on three 40 GB A100 GPUs for 42,000 steps at batch size 140 with AdamW and a cosine learning-rate schedule peaking at 5e-4.

Downstream evaluation used a two-block ResNet head for pairwise secondary structure prediction and a two-layer MLP with GELU activation and hidden size 128 for splice-site classification. On the bpRNA TS0 test set, RNA-xLSTM scored marginally above RNA-FM and below RiNALMo. On ArchiveII — 3,865 sequences across nine families, each held out in turn — it placed second in mean F1 behind RiNALMo and ahead of the 99M-parameter RNA-FM. On the Spliceator GS_1 splice-site benchmark, tested on zebrafish, fly, Arabidopsis, and worm, it trailed both baselines, with mean pooling over token embeddings outperforming the [CLS] representation. Telomerase RNA, the longest family in ArchiveII, was the weakest case for all three models.

#Applications

The most direct use of RNA-xLSTM is as an architectural reference point for anyone weighing non-attention backbones for nucleic acid language models: it establishes what a parameter-matched xLSTM encoder achieves on standard RNA benchmarks under an established training recipe. The released weights and fine-tuning scripts also provide a compact encoder for secondary structure or splice-site tasks at modest compute cost, though researchers who need the strongest accuracy on these benchmarks should use RiNALMo.

#Impact

RNA-xLSTM's contribution is methodological honesty in a subfield where architecture papers overwhelmingly report wins. By holding parameter count, training corpus, tokenization, and downstream heads fixed, it produces one of the cleaner backbone comparisons available for RNA and extends the Bio-xLSTM line of work into a modality it had not covered. Its limitations are acknowledged: the work is a preprint that has not been peer reviewed, only two downstream tasks were evaluated, the 1,024-token pretraining context constrains claims about long-range modeling, and the withheld pretraining code means that stage cannot be reproduced end to end. The paper concludes that broader testing is needed before xLSTM can be said to challenge transformers for RNA.