University of Virginia
Attention-based deep learning model that predicts gene expression from histone modification signals across 56 cell types with interpretable attention scores.
AttentiveChrome is a hierarchical deep learning model designed to predict gene expression levels from histone modification signals in the surrounding chromatin. Developed by the QData lab at the University of Virginia under Yanjun Qi, it was presented at NeurIPS 2017 (Advances in Neural Information Processing Systems, volume 30) under the title "Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin." The model builds directly on an earlier convolutional predecessor, DeepChrome, also from the same group, but replaces the convolutional architecture with a two-level attention mechanism that delivers both superior predictive accuracy and meaningful biological interpretability.
The central problem AttentiveChrome addresses is how to predict whether a gene is highly expressed or lowly expressed in a given cell type purely from the distribution of histone modifications along that gene's locus. Histone marks such as H3K4me3 (associated with active promoters), H3K27me3 (associated with repression), and H3K36me3 (associated with transcription elongation) collectively encode the epigenomic state of chromatin. Prior convolutional approaches could aggregate these signals but could not explain which marks or which genomic positions most influenced the prediction. AttentiveChrome resolves this through two levels of differential attention: one operating across genomic bins within each histone mark, and a second operating across different histone marks jointly. This dual-attention design allows a researcher to ask not only "is this gene predicted to be expressed?" but also "which histone marks and which genomic windows drove that prediction?"
The model was evaluated across 56 different human cell types from the Roadmap Epigenomics Mapping Consortium (REMC), making it one of the first chromatin-based gene expression predictors to be benchmarked at that scale. Its attention scores were shown to provide more interpretable feature attributions than saliency-map-based alternatives, establishing AttentiveChrome as both a practical prediction tool and a framework for understanding chromatin-mediated gene regulation.
AttentiveChrome implements a four-stage computational graph. First, histone modification signals for each of the five marks are independently encoded by a bin-level bidirectional LSTM with hidden size d = 32, producing a sequence of contextual representations across bins. A bin-level soft-attention layer then collapses each mark's sequence into a fixed-length weighted sum, using learned attention energies to emphasize the most informative genomic positions. The resulting per-mark summary vectors are then stacked and processed by a second bidirectional LSTM with hidden size d' = 16 that captures inter-mark dependencies — effectively learning how co-occurring combinations of histone marks jointly constrain expression state. A mark-level soft-attention layer similarly collapses this sequence into a single vector, and a final softmax classification layer predicts binary high/low expression.
The model was trained on Roadmap Epigenomics data using binary expression labels derived from RPKM quantification of RNA-seq in matched cell types, with genes split into equal training, validation, and test partitions of approximately 6,600 samples each per cell type. Across all 56 cell types, AttentiveChrome achieved a mean AUC of 0.8115 and a median AUC of 0.8123, outperforming DeepChrome and its convolutional predecessors across essentially all cell types evaluated. The attention mechanism was validated against saliency maps and found to produce more consistent and biologically coherent feature attributions.
AttentiveChrome is used primarily in regulatory genomics research to understand the relationship between chromatin state and gene expression. Researchers working on epigenome-wide association studies can use the attention weights to identify which histone marks and which genomic loci are most predictive of expression dysregulation in disease contexts. The model has also served as a benchmark baseline for evaluating subsequent chromatin-to-expression models: methods including Chromoformer, TransferChrome, and others report results against AttentiveChrome as the established attention-based predecessor. In studies of cell-type-specific gene regulation, the per-cell-type beta-attention profiles reveal which marks are most informative in each tissue, potentially guiding the design of targeted epigenomic profiling experiments.
AttentiveChrome was among the first deep learning models to demonstrate that attention mechanisms could yield biologically interpretable results in the chromatin-to-expression prediction task, at a time when most interpretability methods for neural networks relied on post-hoc gradient attribution. The NeurIPS 2017 publication brought attention-based epigenomics modeling to the broader machine learning community, and the model has been widely cited as a foundation in the chromatin deep learning literature. It catalyzed a lineage of follow-on models — including Chromoformer, which added 3D chromatin interaction data and transformer-based architectures, and TransferChrome, which added transfer learning across cell lines — that now define the state of the art. A key limitation is that AttentiveChrome operates on a fixed 10 kbp window, making it blind to distal regulatory elements and enhancer-promoter interactions beyond that range. Despite this limitation, its simplicity, interpretability, and established benchmarks continue to make it a useful reference for evaluating newer approaches.