genomicBERT

genomicBERT is a compact genome language model from the Tyagi lab at Monash University and RMIT University, with collaborators at the University of Delhi. It argues a contrarian point in a field that has mostly scaled up: that a carefully built, small DNA vocabulary lets a 89.2M-parameter model match far larger genomic foundation models on downstream classification, at a small fraction of the training cost.

The central design choice is tokenization. Most DNA language models either split sequences into fixed-length k-mers, which forces every possible combination into the vocabulary regardless of biological meaning, or use byte pair encoding, as DNABERT-2 and GENA-LM do. genomicBERT instead trains a unigram tokenizer — a probabilistic subword model built with SentencePiece — directly on the human genome, growing the vocabulary chromosome by chromosome. The resulting 4,096-token vocabulary has a maximum token length of 16, with most tokens between 5 and 9 nucleotides. Because the tokenizer is data-driven rather than rule-based, the same procedure applies unchanged to RNA, and the authors demonstrate this on a pre-miRNA versus other small non-coding RNA classification task.

The model ships inside genomeNLP, a command-line toolkit that packages tokenization, pretraining, fine-tuning, hyperparameter search, and token-attribution interpretation into a single conda-installable pipeline aimed at researchers without a deep learning background.

#Key Features

• Data-driven unigram tokenization: A probabilistic subword vocabulary of 4,096 tokens learned from GRCh38, rather than enumerated k-mers, cutting token counts roughly tenfold relative to 9-mer tokenization on the same data.
• MosaicBERT backbone: ALiBi position biases, FlashAttention, gated linear units, and bfloat16 layer normalization make pretraining fast, and ALiBi lets the model be fine-tuned on sequences longer than the 1,400-nucleotide chunks it was pretrained on.
• Species and biomolecule agnostic: The same pretrained model was fine-tuned on human, mouse, fruit fly, and bacterial datasets, and on RNA fold classification, without task-specific pretraining.
• Token-level interpretability: Layer Integrated Gradients attributions project important tokens back onto sequences, recovering known regulatory motifs such as TATA boxes and the Pribnow box in bacterial promoters, and concentrating on mature miRNA arms in pre-miRNA hairpins.

#Technical Details

genomicBERT follows the MosaicBERT design and totals 89.2M parameters, against 110M for GENA-LM (bert-base) and 117M for DNABERT-2. Pretraining used masked language modeling with a 10% mask ratio over sequences from all 24 human chromosomes of GRCh38, at a maximum sequence length of 256 tokens (roughly 1,400 nucleotides), batch size 2,000, AdamW at a 5.0e-4 learning rate, for 10,000 steps on four NVIDIA A10G GPUs over 15 hours — against 500k steps for DNABERT-2 and 1-2M steps for GENA-LM.

Fine-tuning covered five classification datasets: human TFBS versus coding sequence, human lncRNA versus mRNA, mouse ChIP-seq peaks, fruit fly coding versus non-coding transcripts, and E. coli promoters, all at a 9:1 train-test split with grid-searched hyperparameters. genomicBERT reports accuracy of 0.98, 0.87, 0.96, 0.99, and 0.79 across those tasks respectively, matching DNABERT-2 within a point or two on four of the five and well ahead of GENA-LM, which fails to learn several of the tasks (MCC near zero on three).

#Applications

The model suits labs that need a genomic language model they can actually train and fine-tune on modest hardware — a four-GPU pretraining run of fifteen hours is within reach of a departmental cluster. Typical uses are binary sequence classification problems: identifying transcription factor binding sites, separating coding from non-coding transcripts, calling bacterial promoters, and classifying non-coding RNA families. The genomeNLP wrapper is aimed squarely at biologists, exposing preprocessing, training, and interpretability as command-line steps with interactive metric visualizations, so a group without machine learning engineers can run the whole workflow.

#Impact

genomicBERT's contribution is evidence that vocabulary quality can substitute for parameter count and training compute in genomic language modeling — a useful counterweight in a subfield where model sizes have grown steadily. The genomeNLP toolkit is MIT-licensed on GitHub with a GitLab mirror, conda-installable, and documented with worked case studies, which lowers the barrier for biology groups considerably. The main caveats are settled ones: the work remains a bioRxiv preprint that has not been peer reviewed, pretraining covers only the human genome, benchmarks are limited to binary classification rather than the broader task suites used elsewhere, and no pretrained checkpoint from the paper is distributed alongside the pipeline, so users retrain from scratch.