DNA-xLSTM

A stretch of DNA has two equally valid spellings: the forward strand, and its reverse complement. Nothing in a language model's objective forces it to treat them as the same locus, and models left to learn that equivalence from data alone have been shown to fail at it. The second difficulty is length — regulatory interactions routinely span tens of kilobases, and self-attention charges quadratically for context while keeping a key–value cache that grows with every token decoded.

DNA-xLSTM answers both. It is the genomics member of Bio-xLSTM, a suite of xLSTM variants tailored to biological and chemical sequences by the ELLIS Unit Linz and LIT AI Lab at Johannes Kepler University Linz together with NXAI, presented at ICLR 2025 alongside its siblings Prot-xLSTM for protein families and Chem-xLSTM for SMILES strings. xLSTM revives the LSTM with exponential gating and a matrix-valued memory cell: it parallelises across the sequence during training the way attention does, but decodes recurrently in constant memory with runtime linear in sequence length.

The genomics variant adds reverse-complement equivariant blocks, adopting the two inductive biases introduced by Caduceus — post-hoc conjoining, where the backbone sees both a sequence and its reverse complement and the outputs are summed, and parameter sharing, where equivariance is built into the blocks, the embeddings, and the language-model head. Checkpoints are trained under both causal and masked objectives, so one architecture serves generation and representation learning.

#Key Features

• Strand equivariance by construction: the PH and PS block variants make a prediction invariant to which strand is presented, rather than relying on reverse-complement augmentation to teach the symmetry.
• sLSTM for short context, mLSTM for long: the 1,024-base configurations stack sLSTM blocks, while the 32,768-base configuration uses mLSTM blocks with rotary position embeddings, whose parallel form makes very long training contexts affordable.
• Causal and masked weights from one architecture: causal checkpoints model DNA generatively, while masked bidirectional checkpoints supply embeddings for downstream classifiers.
• Cheap bidirectional pass: for weight-tied layers the reverse-direction cell states are obtained by accumulating the gates backwards — identical to applying the layer twice, but without a second set of stored internal states.
• Eight released checkpoints: at the 1,024-base context, 500k and 2M parameters in each of the causal, post-hoc-conjoined masked, and parameter-shared masked variants; at 32,768 bases, 4M parameters in the causal and post-hoc-conjoined variants only, since parameter sharing is implemented for sLSTM blocks alone.

#Technical Details

Three configurations were trained. DNA-xLSTM-500k has an embedding dimension of 128, five sLSTM blocks, a 1.25:1 up-projection ratio, and a 1,024-token context. DNA-xLSTM-2M has dimension 256, six sLSTM blocks, and a 1:1 up-projection at the same context. The long-range DNA-xLSTM-4M has dimension 256, nine mLSTM blocks, a 2:1 up-projection, rotary position embeddings, and a 32,768-token context. All are pretrained at single-nucleotide resolution on the human reference genome (GRCh38) under both causal and masked objectives, following the experimental protocol of Caduceus and HyenaDNA, whose codebase the implementation is adapted from.

At 2M parameters and 1,024 bases, DNA-xLSTM reaches lower held-out cross-entropy than Transformer++ (a Llama variant), Mamba/Caduceus, and HyenaDNA on both objectives, with the widest margin on masked modelling; at 4M parameters and 32k bases it beats Transformer++ and HyenaDNA and matches Caduceus-4M. Fine-tuned on the 18 Nucleotide Transformer classification datasets, it is the best model under 2M parameters on 12 of 18 tasks, and against the 500M-parameter Nucleotide Transformer v2 it wins 8 of 18 — including splice-site classification at 0.974 accuracy with post-hoc conjoining, and the H3 and H4 histone marks. On the eight Genomic Benchmarks tasks the 500k model is on par with the Mamba baseline overall, taking 5 of 8 against Mamba-PH.

#Applications

The masked checkpoints are drop-in encoders for genomic classification: histone-mark, promoter, enhancer, and splice-site annotation are covered by the fine-tuning scripts shipped with the repository, and the parameter scale keeps a full hyperparameter sweep on modest hardware. The causal checkpoints are generative models of DNA, though no sampling code ships with them — every documented path to a checkpoint runs through the training harness. The repository also carries matched pretraining scripts for Mamba, Caduceus, HyenaDNA, and Transformer++, making it a controlled testbed for comparing sequence-model backbones on genomes.

#Impact

DNA-xLSTM's contribution is evidence that a recurrent architecture with exponential gating and matrix memory is competitive with state space models and transformers on genomic sequence, at matched parameter budgets and under both objectives — and that reverse-complement equivariance transfers cleanly onto a recurrent backbone. The caveats are scale and distribution. The released checkpoints top out at 4M parameters, orders of magnitude below the genomic language models they are benchmarked against, so the results establish an architecture comparison rather than a frontier model; pretraining used the human reference genome alone, without the multi-species corpora later genomic models rely on; and downstream numbers come from fine-tuning rather than zero-shot transfer. The code is Apache-2.0 licensed, but the weights sit in a plain directory index on university web space, carrying no license of their own and no model-hub presence.