GPT2-gene-eng

Genomics has borrowed the vocabulary of linguistics for decades — the genome as a text, motifs as words, chromosomes as chapters — but the analogy rarely produces a working parser. Most DNA language models learn statistical structure directly from nucleotide sequence and leave any notion of a "word" implicit in the tokenizer. GPT2-gene-eng, from Wang Liang at Huazhong University of Science and Technology, takes the analogy at face value. A single GPT-2 is trained from scratch on English prose, protein sequence, and DNA under one shared subword vocabulary; skills such as sentence splitting, paragraph segmentation, and summarization are then taught entirely in English and applied, unmodified, to genomic sequence.

The mechanism is cross-lingual transfer: multilingual models trained with a shared tokenizer generalize an ability learned in one language to others they were never fine-tuned on, and the same effect has been reported between text and biological sequence. After a single classification fine-tune on the English PAWS-X paraphrase task, the model is evaluated on DNA similarity datasets it was never trained for, so the transfer is measured rather than assumed. The segmentation and summarization models built on top are the payload: applied across GRCh38.p14 they turn a chromosome into a browsable hierarchy of volumes, chapters, sections, paragraphs, and sentences, alongside a parallel "English edition" from nearest-neighbour lookup in the shared embedding space.

The checkpoints live on Hugging Face under an organization account named dnagpt, which is an unrelated coincidence of naming. This work shares neither authors, architecture, nor lineage with DNAGPT, the Tencent AI Lab Healthcare genome foundation model, nor with DNAGPT2, the CEITEC compression-oriented GPT-2 family.

#Key Features

• One vocabulary across three languages: A byte-pair encoder trained from scratch on DNA, protein, and English yields roughly 100,000 shared tokens — the precondition for any capability learned in text to reach sequence.
• Transfer that is measured, not assumed: Fine-tuned only on English paraphrase data, the model scores 0.79 to 0.92 accuracy on three held-out DNA similarity benchmarks, and its DNA and English token embeddings visibly converge after fine-tuning.
• Segmentation without DNA labels: The paragraph model learns a <p_end> boundary marker on Wikipedia articles and is then run over raw chromosome sequence; sentence splitting reuses the period token already present in pretraining.
• A hierarchical genome index: Clustering paragraph embeddings twice produces section- and chapter-level structure, giving a multi-level table of contents over a chromosome rather than a flat token stream.
• DNA-to-English rendering: Each DNA token is mapped to its nearest English word in embedding space, yielding a readable parallel edition of the genomic text.

#Technical Details

The backbone is GPT-2 small — 12 transformer layers, 768 hidden units, 12 attention heads — with a 256-token context window and the 100,000-token joint vocabulary, which brings the released checkpoint to 162 million parameters rather than the 117 million of stock GPT-2 small. Pretraining uses roughly 10 GB each of DNA (300–1,000 bp fragments from multiple model organisms, following DNABERT's sampling recipe), protein sequence (Swiss-Prot plus sampled TrEMBL), and English (OpenWebText plus Wikipedia), with 2 GB of each modality reserved for tokenizer training. Training ran for three to five epochs in mixed precision on a single consumer GPU.

Downstream models are fine-tunes of the same backbone: paragraph segmentation on marked-up Wikipedia, summarization on Amazon review title pairs with a TL;DR: prompt and output masking that suppresses non-DNA tokens at generation time. Applied to chromosome 1 of GRCh38.p14, the pipeline splits the chromosome into 25 segments of about 10 Mb; the first segment yields 15,238 paragraphs, which cluster into 503 sections and then 38 chapters. The translation dictionary collapses about 19,000 DNA tokens onto roughly 600 distinct English words, a mapping the paper is explicit is based on embedding-space proximity and carries no semantic claim.

#Applications

The output is infrastructure rather than a predictor. A hierarchical segmentation supports multi-level genome search — locate a chapter, then a paragraph, then a sentence — instead of scanning a flat sequence. Model-generated summaries offer mutation-tolerant identifiers for comparing assemblies, and chapter-level summaries suggest a route to compact, human-inspectable representations of genomic regions. The released checkpoints are usable directly for DNA tokenization, segmentation, and embedding extraction.

#Impact

This is a small, single-author research program, and it earns a catalog entry on its released artifacts and unusual framing rather than benchmark standing. No comparison against established DNA foundation models such as DNABERT or Evo is reported, and the paper states plainly that the segmentation and summarization abilities are hypothesized to transfer, with the genome book offered as an illustration whose biological meaning is not yet established. What it does supply is a complete, Apache-licensed pipeline — tokenizer, base model, three task fine-tunes, and the notebooks that generate the book.