EvoLen

The LMX1B transcription factor recognises the seven-base sequence TAATTAA. A byte-pair encoding tokenizer trained on the human genome for compression splits that site into shorter pieces and merges part of it with the flanking bases, so the binding site is never presented as one unit and the network must reassemble the signal across several tokens. Tokenization is not a neutral preprocessing step for DNA: it fixes which patterns can be represented cheaply, and a frequency objective is indifferent to whether a frequent substring means anything biologically.

EvoLen replaces frequency with evolutionary constraint. Built at UC San Diego and Washington University in St. Louis and published at COLM 2026, it partitions hg38 into conserved, neutral and accelerated pools using phyloP conservation scores, trains a separate BPE vocabulary on each pool, and merges the three under a priority rule that admits tokens shared across all categories first and conserved-specific tokens next. The merged vocabulary is then serialised as a Unigram model whose token scores are the square of token length, so dynamic-programming decoding selects the longest intact segmentation rather than the most frequent one.

The released artifacts are two masked language models pretrained with that tokenizer, evolen-100k and evolen-200k, alongside base-100k and base-200k controls trained with plain whole-genome BPE on identical architecture, corpus and schedule — so any difference downstream is attributable to tokenization alone. That places EvoLen among subword DNA models such as DNABERT-2, GROVER and Nucleotide Transformer, and opposite nucleotide-level models such as Evo and HyenaDNA, which sidestep fragmentation entirely.

#Key Features

• Evolutionary stratification: The genome is binned into 100 bp windows, each scored by mean phyloP and assigned to conserved, neutral or accelerated by a two-tailed z-test at p < 0.1, giving three pools with distinct substitution rates.
• Conservation-prioritised merging: Category vocabularies are combined in a fixed order — shared by all three pools, then conserved-specific, then conserved-and-neutral, then neutral-specific — biasing the vocabulary toward reusable functional units.
• Length-aware decoding: Scoring each token by the square of its length and decoding by dynamic programming rewards longer units when the vocabulary supports them, without forcing long tokens where no coherent multi-base pattern exists.
• Measured motif preservation: At vocabulary size 5,120 it encodes JASPAR 2024 vertebrate motifs of 12 bp or less as single unsplit tokens 27.4% more often than the matched BPE baseline, and raises promoter-enhancer Jensen-Shannon distance from 0.0245 to 0.0361.
• Ablations released as checkpoints: Removing stratification, the priority merge or length-squared scoring each yields a separate published model, so each component's contribution can be re-measured rather than taken on trust.

#Technical Details

Both checkpoints are BERT-base encoders — 12 layers, 768 hidden units, 12 attention heads, 512-token windows, roughly 90 million parameters over a 5,120-token vocabulary — pretrained by masked language modelling on the hg38 human reference at 15% masking, learning rate 4e-5 with 10,000 warmup steps and an effective batch of 768 across eight GPUs. The 200k-step model resumes from the 100k checkpoint under identical hyperparameters. Evaluation covers 56 fine-tuning tasks in 15 groups: GUE, Genomic Benchmarks, Nucleotide Transformer, an 8-way ENCODE SCREEN cCRE task, and a cross-species snATAC-seq brain cell-class task trained on human and evaluated on mouse. Averaged Matthews correlation improves over the matched BPE baseline on 11 of 15 groups, with the largest gains on mouse enhancers (+9.83%), cross-species cell-type prediction (+9.47%), transcription factor binding (+5.70%) and GUE mouse tasks (+4.07%); across the 56 paired tasks the mean change is +0.93 MCC (Wilcoxon signed-rank p = 1.8 x 10^-3). Splice-site tasks, which turn on exact exon-intron boundaries, decline.

#Applications

The checkpoints are drop-in encoders for regulatory genomics: fine-tuning against a labelled sequence set gives transcription factor binding calls, promoter and enhancer classification, cis-regulatory element typing, or chromatin-accessibility-derived cell-class prediction. The tokenizer files ship separately, so groups pretraining their own DNA language model can adopt the vocabulary without using these weights. The cross-species result — trained on human peaks, evaluated on mouse — is the most directly useful for comparative regulatory work.

#Impact

EvoLen is a controlled argument that tokenization carries an inductive bias worth designing, at a moment when most gains in DNA language modelling come from scale. Its improvements are modest and concentrated where the prior is apt — regulatory elements and cross-species transfer — while tasks demanding exact local boundaries do not benefit, and invertebrate and yeast settings fall outside the mammalian conservation scope the vocabulary was built from. The splice-site deficit largely closes under compute-matched 200k-step pretraining, so part of the gap is convergence rather than a fixed property of the tokenizer. These are 90M-parameter research encoders rather than a general-purpose genomic foundation model, and their value lies in the released control and ablation arms: an unusually complete basis for testing whether a different biological prior does better.