RegFM

The same enhancer can drive strong transcription in one cell type and sit silent in another, and the difference is not in the DNA — it is in which transcription factors are present to read it. That asymmetry splits the modeling literature in two. Sequence models such as Enformer, DNABERT-2, Evo 2 and AlphaGenome learn regulatory grammar from DNA alone and return the same answer for every tissue in the body. Single-cell models such as Geneformer and scGPT capture cellular state richly but never see the sequence that state acts upon.

RegFM couples the two halves. It treats transcriptional regulation as a dialogue between cis-regulatory sequence and trans-acting regulators: one transformer encodes a long gene-centered DNA window, a second encodes the activity of 2,103 curated transcription factors (TFs) and chromatin regulators (CRs), and a cross-attention module lets the regulator profile decide which stretches of sequence matter. Crucially, cellular context enters as an input rather than as an axis the model is refit along, so a single pretrained checkpoint can be pointed at a cell type it has never seen given only that cell type's TF/CR expression profile.

RegFM was developed by researchers at Tsinghua University and Yale University, and posted as a preprint in August 2026. It extends the same group's earlier EpiGePT, which paired sequence with TF activity to impute epigenomic tracks, from chromatin signal to transcriptional output and transferable gene-level representations.

#Key Features

• Context is an input, not a refit: A cell type is described to the model by its TF and CR expression, so the same weights transfer to unseen contexts without retraining.
• Long-range gene-centered sequence: Each gene is represented by the ±325 kb window around its transcription start site, wide enough to reach distal enhancers.
• No accessibility data required: Unlike multi-omic regulatory models, RegFM needs only RNA-seq to describe a context, which is far more widely available than paired ATAC-seq.
• Interpretable cross-attention: The attention maps between DNA k-mers and regulators are read directly as candidate CRE-TF-gene relationships rather than being probed post hoc.
• Context-specific gene embeddings: The same gene receives different embeddings in different cell types, which is what carries the representations into downstream tasks.

#Technical Details

The cis-DNA transformer is six LongNet encoder layers with bidirectional multi-head dilated attention at hidden dimension 768, operating on sequence tokenized by a byte-pair-encoding vocabulary of 150,000 k-mers (up to 10 bp) and truncated to 71,680 tokens. The trans-context transformer is 12 layers with 6 heads over a fixed 2,103-token regulator sentence, with TPM values discretized into 256 bins. With the cross-attention module and prediction head, the model has 302 million trainable parameters, pretrained on 8 NVIDIA RTX A6000 GPUs by masked language modeling on each tower followed by supervised expression regression. Training data are 1,018 ENCODE bulk RNA-seq experiments over 281 cell lines and tissues, plus pseudo-bulk profiles for 663 cell types aggregated from roughly 43 million CELLxGENE cells.

Across 56 held-out ENCODE contexts and 3,000 highly variable genes, RegFM reaches a mean Pearson correlation of 0.774 against 0.695 for the strongest sequence baseline; on 133 unseen CELLxGENE cell types it reaches 0.789 against 0.507. A full-scale ENCODE model covering all protein-coding genes (3.6 million training instances) reaches a mean Spearman correlation of 0.610 versus 0.454 for Evo 2. On PBMC leave-one-out across 16 cell types it improves mean Pearson correlation from 0.845 to 0.925 over a model that additionally consumes scATAC-seq peaks. Downstream, embeddings give F1 0.627 on cell-type-agnostic enhancer identification, auROC 0.843 on cell-type-specific enhancer calls, F1 0.843 on bivalent promoter classification, and auROC 0.903 on dosage-sensitivity prediction.

#Applications

The practical draw is annotating regulatory elements and predicting expression in cellular contexts that were never profiled, which matters most for rare cell types and primary tissues where paired chromatin data does not exist; because a context is specified by RNA-seq alone, a lab with a transcriptome can query the model directly. The context-specific gene embeddings also serve as drop-in features: substituted for static gene embeddings in a perturbation-response predictor they sharpen predictions of transcriptional response to genetic perturbation, and they support gene-level prioritization such as flagging dosage-sensitive genes during copy-number variant interpretation.

#Impact

RegFM is a preprint awaiting peer review, and its authors are direct about the limits: byte-pair k-mer tokenization forecloses the single-base resolution that regulatory variant interpretation needs, and the model predicts expression rather than jointly modeling epigenomic signal. What it establishes is narrower and useful — that supplying trans-regulatory state as an input, instead of learning it per dataset, is enough to make a sequence model context-aware, and that doing so beats models given strictly more information in the form of chromatin accessibility. Code and all three pretrained checkpoints are released under the MIT license, leaving the design open to reuse.