seqLens

Genomic language models learn representations of DNA by self-supervised pretraining on large collections of genomes, mirroring the success of protein language models. seqLens is a family of such models developed by the Rahnavard group (omicsEye) at George Washington University and released as a bioRxiv preprint in March 2025. It is designed to produce transferable embeddings of DNA sequence that support downstream genomic classification tasks across bacterial and eukaryotic genomes.

The central design choice in seqLens is its attention mechanism: rather than the standard self-attention used in earlier genomic BERT-style models, it adopts disentangled attention with relative positional encoding, which separates content and position information when computing attention weights. The authors also emphasize the composition of pretraining data, showing that using relevant and taxonomically balanced genomes substantially boosts downstream performance. Together these choices allow the seqLens family to outperform existing genomic language models on the majority of a broad benchmark suite while spanning a wide range of model sizes.

#Key Features

• Disentangled attention: Content and relative position are modeled separately in the attention computation, a design borrowed from DeBERTa-style architectures that improves representation of positional structure in DNA.
• Byte-pair encoding tokenizer: Sequences are tokenized with a 4,096-token BPE vocabulary and a 512-token context window, covering roughly 2.8-3 kilobases of DNA per input.
• Range of model scales: The family includes variants from about 15M to 89M parameters and larger, letting users trade accuracy for compute across deployment settings.
• Data-composition emphasis: The authors demonstrate that pretraining on relevant, taxonomically balanced genomes significantly improves downstream accuracy, not just scaling the number of nucleotides.
• Open weights: Pretrained models are distributed through Hugging Face and GitHub under a CC BY-NC 4.0 license for non-commercial use.

#Technical Details

seqLens uses a transformer encoder with disentangled attention and relative positional encoding, trained with masked language modeling over BPE-tokenized DNA. Two pretraining corpora were assembled: a large prokaryote-heavy dataset of 19,551 reference genomes (over 18,000 prokaryotic) totaling more than 115 billion nucleotides, and a more taxonomically balanced dataset of 1,355 prokaryotic and eukaryotic reference genomes exceeding 180 billion nucleotides. Across a benchmark of 19 genomic tasks, seqLens models outperformed competing approaches on 13, with reported strength on tasks such as plasmid and kingdom classification. Five scale variants (roughly 15M, 23M, 47M, 89M, and larger) are released, allowing comparison of performance against model size.

#Applications

seqLens is intended for researchers working with microbial and metagenomic sequence data who need learned DNA representations for classification and annotation. Typical applications include distinguishing plasmid from chromosomal sequence, taxonomic and kingdom-level classification, and other sequence-labeling tasks where a pretrained genomic encoder can be fine-tuned on modest labeled datasets. The availability of multiple model sizes makes it adaptable to both high-throughput screening and resource-constrained settings.

#Impact

seqLens contributes to the genomic language model landscape by demonstrating that architecture choice (disentangled attention) and pretraining data composition, rather than raw corpus size alone, meaningfully shape downstream genomic performance. Its release of open, non-commercial weights across several model scales, together with a broad 19-task benchmark, provides a reusable resource for the microbial genomics community. As a preprint awaiting peer review, its reported advantages are established on the benchmarks presented by the authors, and the non-commercial license constrains use in industrial pipelines.