GENERator

GENERator is a decoder-only generative language model for eukaryotic DNA, released by GenerTeam — a collaboration led from Alibaba Cloud's Apsara Lab with the Zhongguancun Academy, the University of Science and Technology of China, the Agricultural Genomics Institute at Shenzhen, the two Hong Kong University of Science and Technology campuses, and the Montréal cluster of Mila, Université de Montréal, and HEC Montréal.

The design problem it addresses is the trade-off between resolution and reach. Genomic function is organized over tens of kilobases — promoters, enhancers, introns, and the genes they regulate — but single-nucleotide tokenization makes sequences of that length expensive to model. GENERator resolves this by tokenizing DNA into 6-mers, which lets a fixed 16k-token compute budget cover 96k nucleotides instead of 16k, and pairs that with a functional sequence training strategy that concentrates the corpus on gene-centric regions annotated in RefSeq rather than sampling genomes uniformly.

It is the original release in the GenerTeam line and the direct predecessor of GENERator-v2, which later revisited the tokenization question by recovering single-nucleotide resolution from the same 6-mer vocabulary. GENERator is evaluated three ways: as a pretrained model with no fine-tuning, after task-specific fine-tuning, and — unusually for a genomic language model — through wet-lab validation of sequences it designed.

#Key Features

• 98k-nucleotide context: A 6-mer vocabulary compresses the sequence sixfold, putting whole loci with their regulatory neighborhoods inside a single window at practical cost.
• Alignment-free variant scoring: Zero-shot likelihood differences rank clinical variants without building a multiple sequence alignment, so the method applies to species with no alignment resources.
• Protein-coding sequence generation: Generated coding DNA translates into proteins that fold into structurally plausible models under a structure predictor's confidence scores.
• Prompt-guided regulatory design: Fine-tuning on a massively parallel reporter dataset yields a conditional generator plus an activity predictor, letting users request cis-regulatory elements with a target activity level.
• Experimentally validated designs: Designed enhancers were synthesized and measured, closing the loop between in-silico generation and a real activity readout.

#Technical Details

GENERator is a transformer decoder trained with autoregressive next-token prediction on 386 billion nucleotides of eukaryotic DNA curated from RefSeq and spanning all major eukaryotic lineages. Two scales are released, 1.2B and 3B parameters, both with a 6-mer tokenizer requiring input lengths that are multiples of six. A tokenizer ablation across k-mer lengths one to eight and byte-pair encoding found 6-mers best. On sequence recovery, GENERator-1B reaches 0.52 accuracy against Evo 2-1B at 0.50, and GENERator-3B reaches 0.56 against Evo 2-7B at 0.60, while running roughly nineteen times faster than Evo 2-1B on the same evaluation. On ClinVar variant effect prediction curated by GPN-MSA, GENERator-3B is competitive with Evo 2-7B among alignment-free models, though alignment-based methods retain an edge. Fine-tuned on the DeepSTARR reporter dataset, its activity predictor attains Pearson correlations of 0.71 for developmental and 0.80 for housekeeping cis-regulatory elements. In the design experiment, 12,026 oligonucleotides — including GENERator designs, natural high- and low-activity sequences, and DREAM-designed controls — were assayed by UMI-STARR-seq in Drosophila melanogaster S2 cells; the strongest GENERator-designed housekeeping element exceeded the strongest natural sequence in the DeepSTARR set by 35% and the best DREAM design by 78%. Code and weights are MIT-licensed.

#Applications

The model supports three distinct workflows. Analysts can use it as a frozen encoder or zero-shot scorer for variant interpretation and phylogenetic structure across species where alignments are unavailable. Genomics groups can fine-tune it for promoter classification, enhancer activity regression, and other supervised benchmarks. Synthetic biologists can use the prompt-guided design framework to propose cis-regulatory elements with specified activity profiles for gene therapy cassettes, reporter constructs, and expression tuning — the application the UMI-STARR-seq experiment was built to test.

#Impact

GENERator's lasting contribution is showing that a genomic language model can be pushed past benchmark leaderboards into experimentally verified sequence design, with synthetic super-enhancers outperforming the strongest natural sequences in the reference library. Its efficiency argument — that k-mer tokenization buys long context at a fraction of the inference cost of single-nucleotide models — shaped the follow-on work in GENERator-v2, which addressed the resolution loss that 6-mers introduce. The MIT-licensed release of both checkpoints on HuggingFace made the model a common baseline for later long-context genomic foundation models.