Bio-DTA

Bio-DTA is a multi-modal foundation model that lets a single-cell transcriptome model reason about DNA sequence. Single-cell foundation models such as Geneformer and scGPT represent each gene as a discrete token from a fixed vocabulary, so a gene is the same token whatever its underlying sequence — there is no way to ask how a promoter variant would ripple through the cell's regulatory network. DNA language models such as Enformer have the opposite blind spot: they predict regulatory signal gene by gene from long sequence windows, with no notion of how genes co-regulate one another inside a particular cell.

The mechanism CoSyne Therapeutics introduces to bridge them is dynamic token adaptation (DTA). Multi-modal language models project an image into special tokens prepended to a text sequence, and recent single-cell work borrowed the trick to encode a cell type or drug treatment. DTA generalizes it to every token in the input: each gene's own embedding becomes a learned projection of its DNA-derived representation. A gene token is therefore no longer fixed — change the sequence around its transcription start site and the token changes, and the downstream transformer sees a different cell.

Bio-DTA is the paper's instantiation of that idea, pairing Enformer embeddings with a Geneformer-style BERT encoder. It appeared at ICLR 2025, posted concurrently to arXiv and bioRxiv, and is named only in the paper body, not the title or abstract.

#Key Features

• Dynamic token embeddings: Every input gene's token embedding is computed from its DNA sequence rather than looked up in a fixed table, so in-silico sequence edits propagate directly into the transcriptome model's context.
• One shared adapter for all genes: A single MLP with a softplus activation does the projection, reused across every protein-coding gene rather than fit per gene or dataset.
• Trained end to end by masked language modelling: Adapter and encoder are optimized together on a 15% token-masking objective, so the projection serves the transcriptomic task rather than being aligned post hoc.
• A softer perturbation than gene deletion: Perturbations are applied zero-shot by mutating DNA and re-encoding. Where Geneformer-style analyses simulate a knockout by deleting a gene from the input entirely, altering the whole context, Bio-DTA nudges only the target gene's embedding.

#Technical Details

The encoder follows Geneformer's configuration: a bidirectional BERT with six layers, 256-dimensional embeddings, four attention heads, feed-forward width 512, and an input of 2,048 genes ordered by median-normalized expression. For each protein-coding gene in GRCh38.108, a 196,608 bp window centered on the transcription start site is passed through a public Enformer checkpoint and mean-pooled to a 3,072-dimensional vector, which the adapter projects into the token space. Pretraining used 33,364,242 cells from 265 CZ CELLxGENE Discover datasets, excluding cancer cells and cells expressing fewer than 500 genes, for three epochs — just over seven days on four NVIDIA A10G GPUs.

Evaluation is a single case study on GATA4 and TBX5, interacting congenital heart disease transcription factors, in 103 fetal cardiomyocyte transcriptomes. Randomly mutating 100 bp around the GATA4 start site lowers Enformer's predicted expression; the resulting shift in Bio-DTA's penultimate-layer embeddings is significantly larger for experimentally validated GATA4 and TBX5 ChIP-seq targets than for the rest of the genome (FDR-adjusted p below 0.05 across all five target groups), while housekeeping genes are unaffected. Taking the 50 most-shifted genes as predictions, Bio-DTA leads on precision and recall for direct GATA4, direct TBX5, and co-bound targets; it matches an adapter-free BERT trained on identical data on indirect GATA4 targets and trails it on indirect TBX5 targets. Geneformer, evaluated by in-silico deletion, is outperformed by both and recovers none of the validated GATA4 targets.

#Applications

The intended use is mechanistic: asking how a genetic change to one gene's regulatory sequence propagates through the co-regulation network of a particular cell type, without a perturbation experiment for every hypothesis. That maps onto target identification and variant interpretation in drug discovery, where the question is often which downstream genes a regulatory variant will move in which tissue. DTA is a general recipe rather than a fixed architecture, so the same pattern can carry other per-gene modalities, such as RNA isoform structure or amino acid sequence, into a single-cell model's token space.

#Impact

Bio-DTA's contribution is architectural: a transcriptome model's token vocabulary can be made a function of another modality, turning a discrete gene identifier into a continuous, editable representation. The supporting evidence is deliberately narrow — one transcription factor pair, one cell type, one qualitative readout — and the authors say so, noting that cases where both a perturbation and its cell-type-specific downstream co-regulation are experimentally validated are too rare for broad benchmarking. The paper states two further limits: Enformer was run on the reference genome, and reference-trained DNA models predict eQTL direction poorly; and each gene maps to a single token, capping its genomic context. No code, weights, or checkpoints have been released.