dnaGrinder

Genomic language models are usually pretrained on short windows and then asked to work on long ones. DNABERT-2 caps its pretraining sequences at 128 tokens; a model that has only ever seen 128 positions has no basis for handling a 20,000-token chromosomal region, and in practice such models either truncate the input or run out of GPU memory trying. The alternative the field settled on was scale — the Nucleotide Transformer family reaches 2.5 billion parameters and hundreds of billions of training tokens — which trades the length problem for a cost problem.

dnaGrinder, released as a preprint in September 2024 by Qihang Zhao, Chi Zhang and Weixiong Zhang at The Hong Kong Polytechnic University, takes the other route: keep the encoder small and make length cheap. It pretrains on 12,000 bp windows rather than short fragments, replaces learned positional embeddings with ALiBi so attention degrades smoothly instead of failing outside the trained range, and uses FlashAttention-2 and a SwiGLU feed-forward block without dropout to hold the parameter count to 63.6 million. The result fine-tunes on a workstation card with 12 GB of memory at inputs over 17,000 tokens, and stretches past 140,000 tokens on an 80 GB H100 or A800.

The second contribution is in the data. Rather than choosing between a multispecies corpus and a human corpus with variants, dnaGrinder uses both, and reconstructs the human sequences from phased variant calls so that the model sees real haplotypes rather than a single reference.

#Key Features

• Memory-efficient BPE tokenization: Counting byte-pair frequencies over the full 118 GB corpus at once would require roughly 1.8 TB of RAM, so the tokenizer processes the corpus one split file at a time, updating a running 4,096-token vocabulary as it goes.
• Sequence length warmup in an encoder: Training sequences are ordered from fewest to most tokens, which cuts padding waste and stabilizes gradients — a curriculum previously used for decoder models, applied here to a masked-language-model encoder.
• Extrapolation beyond the training window: With ALiBi, a model pretrained at 12,000 bp classifies 120,000 bp sequences at inference, ten times its training length.
• Both parental haplotypes as augmentation: Each 12,000 bp human window is rewritten with the maternal and the paternal alleles of a randomly chosen 1000 Genomes individual, covering SNVs, indels and structural variants over 50 bp.
• Repeat-aware corpus construction: Soft-masked UCSC assembly annotations are used to drop repetitive regions before window extraction, so roughly half the human genome's duplicated content does not consume training tokens.

#Technical Details

dnaGrinder is a 12-layer encoder-only transformer with 63.6 million parameters, pretrained with masked language modelling at a 15% dynamic mask ratio on eight H100 GPUs — 119,000 steps at batch size 256 and a maximum of 2,314 tokens, totalling about 69 billion training tokens. The corpus combines reference genomes for 794 species from NCBI, the repeat-filtered GRCh38.p14 human assembly, and human windows reconstructed from the 3,202-sample 1000 Genomes phased call set of 73.5 million variants.

Across the 28 GUE classification tasks plus two enhancer tasks from the Nucleotide Transformer suite, dnaGrinder ranks first on 11 and second on 12, with an average score of 73.01 against 70.86 for DNABERT-2 and 68.32 for NT-2500M-multi — roughly 40 times fewer parameters and 29 times fewer FLOPs than the latter. On enhancer type classification it reaches 68.50% accuracy, 4.75 points above the next-best baseline. In a five-species classification task at 120,000 bp, only dnaGrinder and HyenaDNA fit the input on a single GPU, at 100% and 64.22% accuracy respectively.

#Applications

The model targets the standard supervised readouts of regulatory genomics: core promoter and promoter detection, splice site prediction, transcription factor binding in human and mouse, histone mark prediction in yeast, and viral variant classification. Because fine-tuning fits on consumer hardware, it is aimed at labs that want a genomic encoder without cluster access, and its long-context behaviour suits tasks where the relevant regulatory element sits far from the sequence of interest.

#Impact

dnaGrinder is a useful demonstration that most of the gap between small and large DNA encoders comes from tokenization, positional encoding and corpus construction rather than from parameter count. Its practical reach is limited by release status: the preprint is a single arXiv version with no journal publication, its Data Availability section points only at the public NCBI, UCSC and 1000 Genomes sources it drew on, and no code or pretrained weights have been released, so the results cannot be reproduced or the checkpoint reused. The authors also report that further in-domain pretraining, which helped DNABERT-2, degraded scores on 6 of 10 yeast epigenetic tasks for this model.