Nucleotide GPT

Chromosomes are not arranged at random inside the nucleus. Large blocks of chromatin dock against the nuclear lamina, where they are gene-poor and transcriptionally quiet, while other regions associate with nuclear speckles, where genes are highly expressed and splicing is active. Whether that arrangement is written into the DNA sequence itself, or imposed entirely by the protein and epigenetic environment, has been an open question — one that genomic language models are well placed to test, since they read only sequence.

Nucleotide GPT is a decoder-only genomic transformer built by Shae McLaughlin, Sajad Ahanger and Daniel Lim at UCSF to answer it. The model is pretrained from scratch on reference genomes with causal language modeling, evaluated as a general genomic backbone, and then fine-tuned to classify sequences as lamina-associated domains (LADs), speckle-associated domains (SPADs), or neither, using GO-CaRT maps from three cell types of the developing human cortex: radial glia, intermediate progenitors, and postmitotic excitatory neurons.

The design contrasts deliberately with its neighbours. DNABERT, DNABERT-2 and Nucleotide Transformer aggregate DNA into k-mers or byte-pair tokens; HyenaDNA keeps single-nucleotide resolution but reaches long contexts with implicit convolutions rather than attention. Nucleotide GPT keeps both single-nucleotide tokens and full attention, trading context length for interpretable attention maps.

#Key Features

• Nucleotide-level tokenization: Each of A, C, G, T is one token, so regulatory motifs are never split across arbitrary k-mer boundaries.
• Repeat-aware pretraining loss: Softmasked repetitive elements are downweighted by half in the loss rather than excluded, keeping repeat context available without letting it dominate the objective.
• Nuclear compartment prediction: Fine-tuning targets LAD and SPAD association measured in tissue rather than cultured cells, a distinction that matters because roughly half of cell-type-invariant LADs shift under culture.
• Attention as a genome track: Attention scores summed across all layers and heads produce a per-nucleotide signal exported to bedGraph, alignable against ENCODE cCREs, ReMap ChIP-seq density, conservation and JASPAR motif tracks.
• Frozen-feature extraction: A documented embedding call returns per-position representations from the final layer for use in downstream models.

#Technical Details

The architecture is LLaMA-style: 12 transformer layers, model dimension 2,048, eight query and key heads, rotary positional embeddings, RMSNorm before attention and feed-forward blocks, Flash Attention, and an 8,192-token context — roughly 500 million parameters. It is implemented in JAX for TPU, adapted from the Minformer minimal transformer, and pretraining ran on four TPUs for about one day. Pretraining used reference genomes from human, Mus musculus, Macaca mulatta, Danio rerio and Drosophila melanogaster, chunked into 8 kb non-overlapping sequences. Fine-tuning data came from binarizing the genome into 8 kb bins labelled LAD, SPAD, inter-LAD, inter-SPAD, or boundary from GO-CaRT with SEACR domain calling.

On the Genome Understanding Evaluation benchmark the model is competitive rather than leading. Splice site detection reaches 88.01 % accuracy against DNABERT-2's 90.31 %; non-TATA promoter detection reaches 95.57 % against the best reported 97.11 %; human transcription factor binding site prediction reaches 84.60 % on TF-1 and mouse TF-1 reaches 91.59 %, surpassing DNABERT-2's 85.86 %. Histone mark prediction trails the strongest baselines by 3.2 to 15.3 percentage points in MCC. On the target task, LAD classification reaches 77.5 % in radial glia, 74.5 % in intermediate progenitors and 68.9 % in excitatory neurons, averaging 73.6 %; SPAD classification holds near 80 % in all three, averaging 79.4 %. Classification confidence is higher for LADs conserved across cell types (0.540) than cell-type-specific ones (0.476), and UMAP embeddings separate LADs from inter-LADs along a continuous gradient rather than into discrete clusters. A sparse autoencoder is included for interpretability work.

#Applications

The model is aimed at researchers studying three-dimensional genome organization who want to ask what sequence contributes, independent of the epigenetic marks usually used to define compartments. Practical uses include scoring uncharacterized loci for lamina or speckle association, extracting embeddings as features for other chromatin models, and using the attention track to nominate sequence elements at a locus whose compartment assignment is already known. Pretrained checkpoints are archived on Zenodo under CC BY 4.0, at 10,000, 15,000 and 20,000 steps; the code repository carries no license file.

#Impact

The headline finding is that DNA sequence alone predicts nuclear compartment association well above chance in primary tissue, and that the signal is stronger for constitutive LADs than for those that change during neuronal differentiation — consistent with sequence providing a baseline that cell-type-specific factors then modulate. The honest framing of the accuracy matters: 73.6 % on LADs is moderate, and the paper reads it as evidence for a partial sequence contribution rather than a solved prediction problem. The 8 kb window is the main structural limitation, since higher-order organization spans megabases, and the model has not been peer-reviewed.