NucEL

Genomic language models have overwhelmingly borrowed BERT's masked language modeling objective. DNABERT, DNABERT-2, and the Nucleotide Transformer series all mask a fraction of tokens and predict them, which means gradient signal arrives from only the masked positions — typically 15% of the sequence — and the [MASK] token that dominates pretraining never appears during fine-tuning. Both effects cost sample efficiency, and the usual response has been to scale — the largest Nucleotide Transformer is 2.5B parameters.

NucEL, from the Australian National University, takes the other route. It is the first genomic pretraining framework built on ELECTRA's replaced-token detection objective: a small generator proposes substitutions at masked positions, and a larger discriminator classifies every position in the sequence as original or replaced. Supervision therefore covers all tokens rather than the masked minority, and there is no pretrain/fine-tune token mismatch because the discriminator only ever sees real nucleotides. The discriminator is the model carried forward to downstream tasks.

The second design choice is tokenization. Where DNABERT groups bases into overlapping k-mers and DNABERT-2 applies byte pair encoding, NucEL treats each base as its own token. That lengthens sequences but preserves base-level resolution — which matters for single-nucleotide-sensitive tasks and for reading attention maps against known motifs — and the cost is absorbed by a hybrid local-global attention scheme borrowed from ModernBERT. RNAElectra applies the same replaced-token-detection recipe to non-coding RNA.

#Key Features

• Replaced-token detection for DNA: A generator–discriminator pair supplies token-level supervision at every sequence position, rather than only at masked positions, improving pretraining efficiency over masked language modeling at matched compute.
• Single-nucleotide tokenization: A 27-token vocabulary (four bases, seven special tokens, sixteen reserved) preserves base-level precision, against 4,096 entries for 6-mers and roughly 5,000–8,000 for byte pair encoding.
• Hybrid local-global attention: Local attention windows of 128 tokens with a global attention layer every third block, plus flash attention, keep single-base sequences computationally tractable.
• Parameter efficiency: At 93M parameters the discriminator matches or exceeds models 5x to 27x larger across three independent genomic benchmark suites.
• Cross-species transfer from human-only pretraining: Despite seeing only the human reference genome, the model transfers to mouse transcription-factor binding, yeast epigenetic marks, and viral variant classification.

#Technical Details

The generator has 11 transformer layers, hidden size 256, and 8 attention heads; the discriminator — the model carried into downstream tasks — has 22 layers, hidden size 512, and 16 heads, for 93M parameters. Pretraining used the GRCh38/hg38 human reference genome, extracted with a 1,224 bp sliding window at 100 bp overlap and randomly sampled into 1,024 bp segments. Optimization ran for 50 epochs with AdamW at a learning rate of 1e-4, global batch size 192, FP16 mixed precision, and a 1,000-step warmup on 8 NVIDIA A100 GPUs. Downstream adaptation adds a linear head on the discriminator's [CLS] output and fine-tunes end to end.

Evaluation spans three suites. On the Genome Understanding Evaluation (GUE) benchmark, NucEL posts the highest overall average of 75.16 MCC across seven tasks, taking top-1 on core promoter detection and splice site prediction and top-2 on six of seven cross-species tasks — outperforming a multi-species Nucleotide Transformer with 25x more parameters. On Genomic Benchmarks, averaged over five seeds, it reaches 89.9% accuracy against 89.0% for NT2-100M and 88.2% for Caduceus-Ph, best on four of seven tasks. On the revised Nucleotide Transformer benchmark, averaged over ten seeds, it achieves 0.664 mean MCC and state-of-the-art results on 11 of 18 tasks, slightly ahead of the 2.5B-parameter multi-species Nucleotide Transformer at 0.661. Ablations compare single-nucleotide, 6-mer, and byte pair tokenization, and t-SNE analyses of gene biotype embeddings set NucEL against HyenaDNA and DNABERT-2.

#Applications

The model targets the standard regulatory-genomics workload: identifying promoters and enhancers, predicting transcription-factor binding sites, classifying open chromatin, and profiling histone modifications, plus splice-site detection. Its parameter count puts fine-tuning within reach of a single GPU, the practical difference for labs that cannot afford billion-parameter genomic models. Single-base tokenization also makes attention maps interpretable against annotated motifs, and the paper reports cleaner capture of biologically relevant motifs than Nucleotide Transformer — useful when the goal is a mechanistic hypothesis rather than a prediction alone.

#Impact

NucEL's contribution is an objective, not a scale record: it establishes replaced-token detection as a viable and compute-efficient alternative to masked language modeling for genomic sequence, and demonstrates that a 93M-parameter human-only model can match billion-parameter multi-species ones on the field's standard benchmarks. Its ablations on tokenization and masking strategy are directly reusable by anyone designing an ELECTRA-style genomic model. Pretraining is confined to the human genome, so multi-species and long-range regulatory settings lie outside what the reported evaluation establishes. Code and the checkpoint are released under Apache-2.0, but the preprocessed pretraining corpus is not public, so pretraining must be regenerated from GRCh38/hg38 with the repository's code.