GenSyntax

Genome language models have mostly operated on nucleotides, tokenizing DNA directly and learning the statistics of raw sequence. Models in that lineage, such as Evo 2 and the Nucleotide Transformer, tie context length to genome length — a single bacterial chromosome runs to millions of bases, so holding a whole replicon in one window is expensive. GenSyntax, developed at the Shenzhen Institutes of Advanced Technology, CAS with collaborators in China, Japan and Australia, takes the opposite starting point. It begins after annotation and represents a replicon as an ordered list of gene-product descriptors — the free-text product names an annotation pipeline already assigns to each coding feature — concatenated into what the authors call a "genetic paragraph."

Because each token then carries a functional label rather than a base, an entire chromosome compresses into a few thousand tokens of readable text, and a general-purpose LLM backbone can be adapted to it directly. That is what GenSyntax is: a Llama 3.1 8B model continued-pretrained and then instruction-tuned on 49,250 annotated prokaryotic genomes, released alongside GenSyntax-Tiny, a Qwen3-0.6B checkpoint trained the same way for hardware-constrained settings.

The framework is deliberately post-annotation and scoped to prokaryotes. It does not accept raw sequencing reads or unannotated nucleotide sequence; the supported inputs are annotated chromosomes, plasmids, draft genomes, metagenome-assembled genomes and annotated contigs. The work is a bioRxiv preprint, first posted in December 2025 and revised in August 2026, and has not yet been peer reviewed.

#Key Features

• Gene products as the vocabulary: Replacing nucleotide tokenization with product descriptors keeps genome-wide functional context inside a single window and leaves the model input human-readable.
• Four genome-scale tasks from one checkpoint: Plasmid host identification, gene-product disambiguation, circular contig ordering and gene essentiality all run against the same fixed weights, with no per-task retraining.
• Phenotype signal in genome embeddings: Whole-genome embeddings feed classifiers for ten curated BacDive traits, including Gram stain, motility, oxygen tolerance, optimal temperature and cell shape.
• Minimal-genome workflow: An iterative reduction procedure repeatedly scores gene essentiality and strips genes below a threshold, supporting exploratory minimal-genome design.
• Lightweight sibling: GenSyntax-Tiny covers the same four tasks with a 40,960-token default context, extendable to 131,072 tokens with a YaRN configuration.

#Technical Details

The 8B checkpoint is a decoder-only Llama 3.1 transformer — 32 layers, 4,096 hidden dimensions, grouped-query attention with 8 key-value heads, and a 131,072-token position limit. Training proceeds in two stages through LLaMA-Factory: continued pretraining on bare gene-product sequences of the form [chromosomal replication initiator protein DnaA][DNA polymerase III subunit beta]…, followed by Alpaca-style supervised fine-tuning on task instructions. The reported runs used 40 NVIDIA H100 80 GB GPUs across five nodes in BF16 with DeepSpeed ZeRO-3 and CPU offload.

Evaluation uses released held-out sets: 1,000 plasmids scored for host accuracy from class through strain, 500 four-option and 500 eight-option gene-product disambiguation items, 500 records each for three-, four- and five-contig circular ordering scored as exact cyclic rotations, and 1,000 gene essentiality records reported with class-specific and macro precision, recall and F1. Every evaluator applies 100 sample-level percentile bootstrap replicates with a two-sided 90% confidence interval and keeps unparseable generations in the denominator as errors rather than discarding them. Inference is served through vLLM and fits on a single 24 GB GPU at standard context.

#Applications

The immediate users are microbial genomics and synthetic biology groups working downstream of an annotation pipeline. Plasmid host identification supports mobile-element and antimicrobial-resistance surveillance; gene-product disambiguation helps resolve ambiguous or competing functional calls; contig ordering assists in finishing draft and metagenome-assembled genomes; and essentiality prediction offers a cheap triage step before knockout or knockdown experiments. The embedding route extends this to phenotype-from-genome classification, and the minimal-genome loop gives chassis designers a way to nominate dispensable gene sets computationally before committing to construction.

#Impact

GenSyntax's main contribution is a change of substrate: by treating curated functional annotation as the modeling unit, it makes whole-replicon reasoning tractable for off-the-shelf LLM backbones and yields predictions that can be traced back to named gene products. That complements rather than replaces nucleotide-level genome models, which remain the option when annotation is unavailable or when sub-genic resolution matters. Practical adoption depends on upstream annotation quality, and the scope is prokaryotic throughout. The code is MIT-licensed and the evaluation data is released under Apache 2.0, while the 8B weights carry the Llama 3.1 Community License inherited from the backbone; the authors also run a public web interface for interactive single and batch prediction.