Gene42

Regulatory elements can act on genes hundreds of kilobases away, so the useful context window for a genomic model is far longer than a transformer's quadratic attention cost comfortably allows. The field's answer has largely been to abandon dense attention: HyenaDNA uses implicit long convolutions, Caduceus builds on Mamba state-space blocks, and Evo mixes attention with data-controlled convolutions. Dense-attention models stayed short — 6,000 bp for Nucleotide Transformer, 36,000 bp for GENA-LM.

Gene42, from the Inception Institute of Artificial Intelligence with M42 and Cerebras Systems, tests whether that retreat was necessary. It is a family of decoder-only, LLaMA-style genomic language models that keeps full dense self-attention while reaching a 192,000 bp context at single-nucleotide resolution, obtained by continued pretraining from a 4,096 bp base model rather than by training long from the start. It is part of Inception's Omics42 platform, alongside the Prot42 protein and Chem42 chemical language model families.

The family spans a 500M-parameter base model, Gene42-B, and a 1.1B-parameter Gene42-L released at several context lengths. All are evaluated as pretrained backbones — embeddings feeding a classifier, or full fine-tuning — across biotype classification, regulatory element identification, chromatin profiling, variant pathogenicity and species classification.

#Key Features

• Dense attention at 192 kbp: Every token attends to every other across the full window, without the convolutional or state-space approximations that competing long-context genomic models rely on.
• Incremental context extension: The context window grows through continued pretraining in stages — 4,096 to 8,192, 16,384, 32,768, 65,536, 131,072 and finally 192,000 tokens — by lowering the RoPE base frequency at each step so distant positions retain influence.
• Single-nucleotide tokenization: A character tokenizer over A, C, G and T avoids the k-mer and byte-pair schemes that split motifs across tokens, preserving the resolution needed to score individual variants.
• A family, not a single checkpoint: 500M and 1.1B parameter models at several context lengths let downstream users match window size to task, since the longest window is not always the best performer.
• Multi-species pretraining option: A variant adds 8.1 billion tokens from 14 vertebrate species to the human corpus, improving several regulatory element benchmarks.

#Technical Details

Gene42 follows the LLaMA recipe — rotary position embeddings, SwiGLU activations, RMSNorm pre-normalization — as an autoregressive decoder. Gene42-B has 16 layers and hidden size 1,408; Gene42-L has 24 layers, hidden size 2,048 and 32 attention heads. Pretraining used the GRCh38 assembly split 99/1, giving 3.5 million sequences of 4,096 tokens, or 14.5 billion nucleotides, and ran on Cerebras CS-2 systems with AdamW. Perplexity is 1.61 at a 65,536 bp context, corresponding to 0.789 next-base reconstruction accuracy, rising to 1.85 at the full 192,000 bp window.

On Ensembl biotype classification from frozen embeddings, Gene42-L reaches an F1 of 0.782 at 65,536 bp, against 0.759 for the 2.5B-parameter Nucleotide Transformer and 0.709 for HyenaDNA-medium. Across the eight Genomic Benchmarks datasets it averages 89.3% top-1 accuracy versus 88.5% for HyenaDNA and 86.5% for Caduceus, leading on five. On the 18-task Nucleotide Transformer Benchmark, the 500M model tops 8 datasets, matching the 2.5B multispecies Nucleotide Transformer, with an epigenetic-marks average MCC of 0.627. Fine-tuned on 64,500 ClinVar variants, it reaches an AUC-ROC of 0.931 against 0.892 for GENA-LM. On DeepSEA chromatin profiling it leads on transcription factor binding (median AUC 0.967) and DNase hypersensitivity (0.934) but trails on histone marks (0.839 versus 0.856). Five-way species classification reaches 99.5% at 32,768 bp.

#Applications

Gene42 targets the genome-interpretation tasks that benefit from seeing an entire regulatory neighborhood at base resolution: linking distal enhancers and promoters to the genes they act on, predicting chromatin state from sequence, and classifying variants of uncertain significance in clinical genetics. The frozen-embedding results indicate it can also be used as a feature extractor with a lightweight classifier when fine-tuning budgets are limited. Its developers position it within a broader precision-medicine effort spanning genomics, proteins and chemistry.

#Impact

Gene42's main contribution is a negative result about a widely held assumption: the field moved to state-space and long-convolution architectures partly because dense attention was thought infeasible at genomic scale, and Gene42 shows that staged context extension makes 192 kbp windows trainable while matching or beating those alternatives on short- and long-range benchmarks alike. Practical uptake is limited by availability. The paper directs readers to Inception's Hugging Face organization for the models, but no Gene42 checkpoint is listed there and no source repository accompanies the release, so the results have not been independently reproduced. The work is also a preprint that has not been peer reviewed, and its ablations do not isolate how much of the benchmark gains come from long context as opposed to model scale — the 500M 4,096 bp model is the strongest Gene42 configuration on several tasks.