Seoul National University
Transformer-based model for predicting gene expression from histone modifications, incorporating 3D chromatin interaction data and large genomic windows to capture distal regulatory effects.
Chromoformer is a transformer-based deep learning architecture for quantitative prediction of gene expression from histone modification profiles, developed by Dohoon Lee, Jeewon Yang, and Sun Kim at Seoul National University's Interdisciplinary Program in Bioinformatics and published in Nature Communications in November 2022. It addresses a fundamental limitation shared by earlier chromatin-to-expression models such as DeepChrome and AttentiveChrome: those models could only examine histone mark signals within narrow linear windows around promoters, typically spanning 10 kilobases or fewer, and therefore lacked any view of distal regulatory elements or the three-dimensional organization of chromatin in the nucleus.
Chromoformer overcomes this limitation through two conceptual advances. First, it extends the genomic window examined around each transcription start site from ~10 kbp to 40 kbp, allowing the model to capture proximal cis-regulatory elements such as upstream enhancers and downstream elongation markers. Second, and more distinctively, it incorporates pairwise promoter–enhancer interactions derived from promoter-capture Hi-C experiments, allowing the model to access histone modification signals at distal genomic regions whose chromatin physically contacts the promoter in 3D nuclear space. These interactions can span tens to hundreds of kilobases along the linear genome but are functionally relevant because chromatin looping brings them into close spatial proximity with the promoter.
The model's three-module transformer architecture conceptually decomposes gene regulation into a hierarchy: local chromatin context at the core promoter, pairwise regulatory contributions from looping elements, and the collective regulatory state imposed by the full set of 3D contacts. By learning representations at each of these scales simultaneously, Chromoformer substantially outperforms prior models across all tested cell types and prediction tasks. Pretrained model weights for 11 cell types are publicly available on Figshare, making the model accessible without requiring GPU-intensive retraining from scratch.
Chromoformer's architecture reflects a deliberate decomposition of gene regulation into three levels of organization. The Embedding Transformer processes a 40 kbp window of histone modification signals around the transcription start site using standard self-attention to extract local promoter context. The Pairwise Interaction Transformer uses encoder-decoder attention to integrate each putative cis-regulatory element (pCRE) — identified from Hi-C contact maps — with the core promoter representation, incorporating normalized interaction frequency as an additive bias to the attention logits. The Regulation Transformer then applies gated self-attention across all pCRE representations to synthesize a collective regulatory state vector, which feeds into the final prediction head.
Training data derived from the Roadmap Epigenomics Project and ENCODE, spanning 18,955 RefSeq-annotated genes with complete ChIP-seq, RNA-seq, and promoter-capture Hi-C profiles in each of the 11 cell types. Benchmarking against DeepChrome, AttentiveChrome, HM-CRNN, and DeepDiff showed consistent improvement across all cell types and all three prediction tasks (binary ROC-AUC, regression Pearson r, and fold-change accuracy). A key empirical finding was that performance continued to improve as the input window expanded to 40 kbp, whereas competing models showed no benefit or performance degradation beyond 10 kbp windows — demonstrating that the transformer architecture's global attention is necessary to exploit the larger context.
Chromoformer is well-suited for studies that require quantitative predictions of gene expression from epigenomic profiling data, particularly when Hi-C or other 3D chromatin contact data are available. Researchers studying cell-type-specific gene regulation can use the Pairwise Interaction Transformer's attention weights to identify which distal regulatory elements are predicted to contribute most to promoter activity, providing testable hypotheses for CRISPR perturbation experiments. The model is also applicable to the study of disease-associated chromatin remodeling: by comparing Chromoformer predictions under normal versus perturbed histone mark profiles, researchers can quantify the predicted transcriptional impact of epigenomic changes observed in cancer, development, or in response to drug treatment. The three-task design makes it compatible with both binary classification workflows and continuous quantitative genomics pipelines.
Chromoformer established that transformer architectures with explicit 3D chromatin interaction inputs meaningfully advance the state of the art in chromatin-to-expression prediction, beyond what is achievable with larger linear windows alone. The Nature Communications publication and the open availability of pretrained weights have facilitated adoption across epigenomics labs. The model's mechanistic interpretability — particularly the finding that its attention naturally learns H3K36me3 signals 4–6 kbp downstream of transcription start sites, consistent with the known biology of transcriptional elongation — reinforced the broader argument that attention-based architectures can recover biologically meaningful regulatory grammar. A key limitation is the dependence on promoter-capture Hi-C data, which is not available for all cell types and requires substantial experimental effort to generate. Future extensions may incorporate predicted Hi-C contact maps or use alternative 3D data sources such as ChIA-PET or Micro-C to broaden applicability.