METAGENE-1

Genomic language models are almost always trained on curated reference genomes — a human assembly, a panel of model organisms, a collection of sequenced bacterial species. That choice bakes in an assumption that breaks precisely where biosurveillance needs it to hold: the sequence that matters during an emerging outbreak is the one nobody has catalogued yet, arriving as a noisy 150-base read from a mixed environmental sample.

METAGENE-1 inverts the recipe. It is a 7-billion-parameter autoregressive transformer pretrained on over 1.5 trillion base pairs of DNA and RNA sequenced directly from human wastewater, a corpus assembled over six months by the Nucleic Acid Observatory from municipal influent across California and Missouri and never previously used for model training. Because wastewater carries bacteria, viruses, human cells, and other eukaryotes together in whatever proportions the population actually sheds, the pretraining distribution is the metagenome as it is encountered rather than as it has been assembled. The model was built with Prime Intellect and researchers at the University of Southern California, and released openly in January 2025.

The design targets the short-read regime this data lives in. Reads are mostly 100 to 300 base pairs after trimming, so a 512-token context is ample and there is no need for the long-context architectures pursued by HyenaDNA and Evo — a standard dense decoder-only transformer is the efficient choice, and it inherits the whole tooling ecosystem built around that shape.

#Key Features

• Wastewater pretraining corpus: Over 1.5 trillion base pairs from deep metagenomic sequencing of municipal influent, spanning the full mixture of organisms present rather than a curated species list.
• Metagenomic byte-pair encoding: A BPE tokenizer trained on 150 million reads sampled uniformly at random, with a 1,024-token vocabulary, so token length adapts to recurring motifs instead of being fixed by a k-mer choice, and novel sequences remain tokenizable.
• Pathogen detection under distribution shift: Fine-tuned with LoRA, it separates human-pathogen from non-pathogen reads across four sequencing deliveries that differ in collection site, date, and pipeline.
• Loss-based anomaly detection: Length-normalised cross-entropy cleanly separates in-distribution metagenomic reads from human, mouse, and random sequences, giving a detector for out-of-distribution material with no classifier at all.
• Open release with a stated safety rationale: The weights and their quantised variants, the four evaluation datasets, and the Gene-MTEB and DGEB benchmark repositories all carry Apache-2.0, and the paper argues the release case explicitly on capability grounds.

#Technical Details

The architecture follows Llama-2-7B: 32 layers, 32 attention heads, embedding size 4096, intermediate size 11008, RMSNorm, rotary position embeddings, no bias terms, z-loss regularisation, and a 512-token context with attention masked between packed reads. Pretraining ran for one epoch over the 1.5-trillion-base-pair corpus on four nodes of eight H100 SXM5 GPUs, reaching 40% model FLOPs utilisation under PyTorch FSDP hybrid sharding despite a 40 GB/s Ethernet interconnect, with a global batch of 30,720 and a cosine schedule from a 6e-4 peak.

On the paper's four-way pathogen detection benchmark, METAGENE-1 averages 92.96 MCC against 87.92 for DNABERT-2, 87.02 for DNABERT-S, and 82.43 for the 2.5B multispecies Nucleotide Transformer. On Gene-MTEB, a genomic embedding benchmark introduced alongside the model, zero-shot mean-pooled embeddings reach a global average of 0.590 against 0.545 for the strongest Nucleotide Transformer variant. On the 28 tasks of the GUE benchmark it takes the top score on 13, against 7 for DNABERT-2, while lagging on core promoter detection — a gap the authors attribute to the pretraining mixture containing little curated eukaryotic promoter sequence.

#Applications

The intended use is wastewater biosurveillance: scoring raw sequencing reads for pathogen content, flagging anomalous material that matches nothing in the training distribution, and producing embeddings that downstream clustering or retrieval can run over at scale. Beyond public health, the GUE results show the model transfers to conventional genomics classification, and the small vocabulary and short context make LoRA fine-tuning on a single GPU realistic without a large compute budget.

#Impact

METAGENE-1 establishes environmental metagenomic sequence as a distinct pretraining substrate rather than a downstream application of genome-trained models, and it ships two reusable benchmarks, Gene-MTEB and DGEB, alongside the weights. The release is partial, though. Apache-2.0 covers the model repositories, the evaluation datasets, and the benchmark code; the pretraining repository carries no licence at all and its README describes itself as still being organised for release, and the wastewater corpus behind the model has not been published, so the pretraining run cannot be reproduced. The paper devotes a section to the decision to open-source, arguing that a 512-token context limits utility for sequence design and that the pandemic-preparedness benefit outweighs the misuse risk at this capability level, while advocating formal safety assessment before larger models of the same kind are released. It remains a preprint, and no field deployment results are reported.