CENO

DNA carries function across a continuum of length scales — single-nucleotide grammar, transcription-factor motifs, regulatory neighborhoods, and chromatin domains spanning hundreds of kilobases. Most genomic language models pick one end of that continuum: encoders such as DNABERT-2 and Nucleotide Transformer classify short windows, while autoregressive models such as Evo 2 generate sequence. CENO, from Shanghai AI Laboratory and collaborators, is built around the argument that a single likelihood interface can serve both. The authors call the result a genomic world model: one autoregressive model that maintains nucleotide-resolution state over megabase context, scores counterfactual mutations, conditions on homologous sequence evidence, and generates candidate sequences.

What makes this practical is architectural. CENO interleaves Mamba-2 sequence-mixing blocks, sparse attention, and mixture-of-experts capacity in one causal backbone: the recurrent blocks carry long-range state cheaply, the attention layers supply the content-based lookup that convolutional models such as HyenaDNA lack, and the MoE layers add capacity without proportional compute per token.

CENO was released as a bioRxiv preprint on 30 July 2026 and has not yet been peer reviewed. Fifteen checkpoints are on HuggingFace — four base sizes across three context stages, plus the MSA post-trained CENO-P family — all loadable via the standard from_pretrained(..., trust_remote_code=True) path under Apache-2.0.

#Key Features

• Hybrid recurrent-attention-MoE backbone: Mamba-2 blocks (state dimension 128), sparse attention, and eight-expert MoE layers with top-2 routing share one causal stack, giving 1B total parameters but only about 400M active per token.
• Staged context curriculum to 1M tokens: 8,192-token cross-domain pretraining is followed by 131,072- and 1,048,576-token continuation on complete eukaryotic genomes, so long-range ability is added on top of settled local grammar rather than traded against it.
• Evolution-conditioned variant scoring: CENO-P post-trains the backbone on packed real multiple-sequence alignments, interleaving row-local and fusion layers, and scores variants by the reference-minus-mutant likelihood delta — the same zero-shot interface used in pretraining, in the spirit of GPN-MSA.
• Zero-shot gene-scale continuation: given the first 30%, 50%, or 80% of a withheld gene, the model regenerates the remainder across eukaryotic, bacterial, and archaeal species, with recovery improving with scale and long-context training.
• Long-context inference throughput: sustained generation out to 131,072 tokens runs at 627 tokens/s/GPU for the 1B model versus 67 for Evo 2 7B, which ran out of memory entirely at the 131k prefill setting.

#Technical Details

CENO was pretrained on OpenGenome2, the same corpus used by Evo 2, spanning GTDB/IMG prokaryotic, metagenomic, viral, organelle, and eukaryotic genomes plus transcript, splice, and regulatory windows, with Evo 2's safety exclusions retained. A constrained RegMix search set the mixtures: a near-balanced 51%/49% prokaryotic-versus-eukaryotic Stage I recipe, shifting to 32%/68% for the long-context stages. The 300M, 600M, and 1B models used 9, 20, and 38 layers and short-context budgets of 1.5T, 3.3T, and 6.0T tokens, followed by 500B tokens each at 131k and 1M context. Tokenization is byte-level with a 512-entry vocabulary.

On a frozen-backbone linear probe for TAD boundaries, mean AUROC rises from 0.767 at 8k context to 0.960 at 131k for the 600M model in human cell types, with comparable gains in mouse. MSA post-training lifts BRCA1 variant AUPRC by 45.3%, 46.4%, and 13.5% over the matched pretrained checkpoints at 300M, 600M, and 1B. On the GPN-Star human-disease panel, the best CENO-P places second on COSMIC (AUPRC 0.270) and third on OMIM (0.657), and it ranks first on all five cross-species conservation panels, with Fisher odds ratios up to 18.30 in D. melanogaster. On TraitGym the CENO-1B checkpoints cluster at 0.595-0.611 mean AUROC, behind Evo 2 40B.

#Applications

CENO suits work that needs one model across a locus rather than a window: scoring noncoding variants whose causal element sits far away, probing chromatin organization from frozen representations, and reconstructing gene-scale sequence in sparsely annotated species. As a downstream case study rather than a zero-shot capability, the authors fine-tune CENO into a cell-type-specific accessibility oracle for mouse motor cortex — trained on 440,993 consensus scATAC-seq peaks and 73,323 cell-type-specific regions, reaching Pearson r = 0.81 — then couple it to a conditionally fine-tuned generator optimized by reinforcement learning to design candidate enhancers.

#Impact

CENO's contribution is less a benchmark sweep than a demonstration that hybrid recurrent architectures make megabase genomic context economically usable: an order-of-magnitude throughput advantage over comparable Evo 2 configurations, at parameter counts 7-40x smaller, with competitive variant-effect performance. The authors are explicit about what this does not settle. Evo 2 40B still leads TraitGym and bacterial gene recovery, downstream performance is not monotonic in model size, MSA post-training depends on real homologous rows, and designed enhancers are candidates for MPRA or AAV validation rather than validated elements, since reinforcement learning can exploit weaknesses in the learned oracle. With Apache-2.0 code and 15 open checkpoints, the family is a substrate other groups can probe and extend.