GTA

The expression quantitative trait loci that regulate a gene are scattered across roughly a megabase around its transcription start site, yet Enformer — the leading sequence-to-expression model GTA benchmarks against — takes in only 200 kb, and studies find models in this family capture promoter grammar while largely missing distal enhancers. Widening the window is not just a matter of a bigger network: nucleotide-level feature extractors get harder to learn as the context grows.

GTA (Genetic sequence Token Alignment), from the Division of Artificial Intelligence in Medicine at Cedars-Sinai Medical Center with statisticians at UCLA, answers this by not learning a sequence encoder at all. A frozen Sei compresses each non-overlapping 4,096-base bin of a one-megabase window into 33 named regulatory sequence classes — enhancers, promoters, transcription, heterochromatin and cell-type-specific variants of each. A learned cross-attention module maps those bins onto "text prototypes" assembled from the token embeddings of a frozen Llama3-8B, so they can pass through the language model as if they were words; a selector and a linear head then read out one expression value.

Both backbones stay frozen: this is model reprogramming, not fine-tuning. Only the alignment module, the selector and the output head are trained, and the model trains on a single A100 GPU. GTA appeared on arXiv in October 2024 and as a poster at the AI for Nucleic Acids workshop at ICLR 2025.

#Key Features

• Frozen backbones, a thin trained bridge: Neither the genomic feature extractor nor the language model is updated. Only the cross-attention aligner, the selector and the linear regression head learn, which is what makes a megabase of context affordable.
• Named features instead of opaque embeddings: Because Sei's sequence classes carry descriptive names, half of GTA's 100 text prototypes are built from the tokens of those names rather than the full vocabulary, and those seeded prototypes attract markedly higher cross-attention.
• Context that scales by bins, not parameters: Sei featurization is position-agnostic, so extending the input window adds bins rather than weights. Five variants were trained, spanning roughly 200 kb to 1,000 kb.
• Gene annotations as prompts: NCBI Gene records — symbol, full name, aliases and the RefSeq summary — are tokenized ahead of the sequence bins, conditioning the prediction on curated human knowledge within GTA's own trained pipeline.
• Learned selection instead of fixed cropping: Where Enformer crops fixed regions from both ends of its input, GTA uses a linear layer to pick the 25 final activations that feed the regression head, and those choices can be read off directly.

#Technical Details

Each gene is represented by 251 bins of 4,096 bases — 1,028,096 in total — centred on its MANE transcription start site in GRCh38, run through Sei and reduced to 33 sequence classes once the "low signal" classes are dropped. Alignment uses 8 cross-attention heads with key dimension 32, projected into Llama3-8B's 4,096-dimensional embedding space against 100 text prototypes. Prompt and bins together are truncated to 2,048 tokens using rotary position embedding scaling, and an ablation found performance peaks at 8 of Llama3-8B's 32 transformer blocks, so the reported models retain only those. Training minimizes mean squared error with Adam.

The benchmark predicts median expression across 421 Geuvadis individuals in GM12878 lymphoblastoid cells: 15,087 genes split 80/20 into training and validation, with 2,850 held out for evaluation. GTA reaches a Spearman correlation of 0.6527 ± 0.0241, against 0.5814 ± 0.0293 for Enformer, 0.5234 for Basenji, 0.5170 for ExPecto and 0.2984 for Xpresso — and unlike those baselines, whose training sets overlap the evaluation genes, GTA's does not. Accuracy across the five context lengths is not monotonic (0.6259 at 51 bins, 0.6078 at 151), so the megabase result is the best of five trained variants rather than a clean scaling curve.

#Applications

GTA suits regulatory genomics questions where distal sequence matters: ranking candidate enhancers by the attention they draw for a gene, probing which parts of a megabase window a model relies on, and predicting expression for genes unseen in training. The attention analysis is the more transferable product — one head attends almost exclusively to B-cell, monocyte and T-cell enhancer classes, matching the lymphoblastoid training data. Emitting a single scalar for one cell type, its value is methodological rather than as a general-purpose predictor.

#Impact

GTA's contribution is methodological: it shows that a general-purpose language model can be reprogrammed rather than retrained for a genomics regression, and that semantically named features are a better interface to it than raw embeddings. It also decouples receptive field from model size, a constraint that has shaped this lineage from Basenji through Enformer to wider-context successors such as Borzoi. The evidence base is narrow — one cell type, a held-out gene split rather than transfer to an external dataset, and no released code or weights — so the result stands as a proof of concept for cross-modal adaptation rather than a model to deploy.