AttentiveChrome

AttentiveChrome is a hierarchical deep learning model designed to predict gene expression levels from histone modification signals in the surrounding chromatin. Developed by the QData lab at the University of Virginia under Yanjun Qi, it was presented at NeurIPS 2017 (Advances in Neural Information Processing Systems, volume 30) under the title "Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin." The model builds directly on an earlier convolutional predecessor, DeepChrome, also from the same group, but replaces the convolutional architecture with a two-level attention mechanism that delivers both superior predictive accuracy and meaningful biological interpretability.

The central problem AttentiveChrome addresses is how to predict whether a gene is highly expressed or lowly expressed in a given cell type purely from the distribution of histone modifications along that gene's locus. Histone marks such as H3K4me3 (associated with active promoters), H3K27me3 (associated with repression), and H3K36me3 (associated with transcription elongation) collectively encode the epigenomic state of chromatin. Prior convolutional approaches could aggregate these signals but could not explain which marks or which genomic positions most influenced the prediction. AttentiveChrome resolves this through two levels of differential attention: one operating across genomic bins within each histone mark, and a second operating across different histone marks jointly. This dual-attention design allows a researcher to ask not only "is this gene predicted to be expressed?" but also "which histone marks and which genomic windows drove that prediction?"

The model was evaluated across 56 different human cell types from the Roadmap Epigenomics Mapping Consortium (REMC), making it one of the first chromatin-based gene expression predictors to be benchmarked at that scale. Its attention scores were shown to provide more interpretable feature attributions than saliency-map-based alternatives, establishing AttentiveChrome as both a practical prediction tool and a framework for understanding chromatin-mediated gene regulation.

#Key Features

• Two-level hierarchical attention: A bin-level attention mechanism (alpha attention) learns position-specific importance weights across genomic bins for each histone mark, while a histone-mark-level attention mechanism (beta attention) learns the relative contribution of each mark to the expression prediction. Both levels are trained jointly end-to-end with the expression classification objective.
• LSTM-based sequence encoding: Bidirectional LSTM modules at each level of the hierarchy encode the sequential structure of histone modification signals along the gene locus, allowing the model to capture positional dependencies that purely pooling-based methods miss.
• Five-mark input representation: The model is trained on five core histone modifications — H3K27me3, H3K36me3, H3K4me1, H3K4me3, and H3K9me3 — measured in 100 bp bins across a 10 kbp window centered on each gene's transcription start site, representing a compact yet information-rich view of the epigenomic landscape.
• Broad cell-type coverage: AttentiveChrome is trained and evaluated on 56 independent cell-type datasets from REMC, each containing approximately 19,802 gene examples. This scale ensures that results reflect genuine generalization across diverse cell types rather than overfitting to a single tissue.
• Interpretable mark-level weights: The beta-attention weights over histone marks can be aggregated and visualized across all genes to reveal which chromatin features the model considers most informative for a given cell type, providing a quantitative complement to traditional ChIP-seq enrichment analyses.
• Open source with reproducible benchmarks: Code, data, and pre-trained models are released on the QData GitHub repository, enabling direct comparison with subsequent methods and integration into custom analysis pipelines.

#Technical Details

AttentiveChrome implements a four-stage computational graph. First, histone modification signals for each of the five marks are independently encoded by a bin-level bidirectional LSTM with hidden size d = 32, producing a sequence of contextual representations across bins. A bin-level soft-attention layer then collapses each mark's sequence into a fixed-length weighted sum, using learned attention energies to emphasize the most informative genomic positions. The resulting per-mark summary vectors are then stacked and processed by a second bidirectional LSTM with hidden size d' = 16 that captures inter-mark dependencies — effectively learning how co-occurring combinations of histone marks jointly constrain expression state. A mark-level soft-attention layer similarly collapses this sequence into a single vector, and a final softmax classification layer predicts binary high/low expression.

The model was trained on Roadmap Epigenomics data using binary expression labels derived from RPKM quantification of RNA-seq in matched cell types, with genes split into equal training, validation, and test partitions of approximately 6,600 samples each per cell type. Across all 56 cell types, AttentiveChrome achieved a mean AUC of 0.8115 and a median AUC of 0.8123, outperforming DeepChrome and its convolutional predecessors across essentially all cell types evaluated. The attention mechanism was validated against saliency maps and found to produce more consistent and biologically coherent feature attributions.

#Applications

AttentiveChrome is used primarily in regulatory genomics research to understand the relationship between chromatin state and gene expression. Researchers working on epigenome-wide association studies can use the attention weights to identify which histone marks and which genomic loci are most predictive of expression dysregulation in disease contexts. The model has also served as a benchmark baseline for evaluating subsequent chromatin-to-expression models: methods including Chromoformer, TransferChrome, and others report results against AttentiveChrome as the established attention-based predecessor. In studies of cell-type-specific gene regulation, the per-cell-type beta-attention profiles reveal which marks are most informative in each tissue, potentially guiding the design of targeted epigenomic profiling experiments.

#Impact

AttentiveChrome was among the first deep learning models to demonstrate that attention mechanisms could yield biologically interpretable results in the chromatin-to-expression prediction task, at a time when most interpretability methods for neural networks relied on post-hoc gradient attribution. The NeurIPS 2017 publication brought attention-based epigenomics modeling to the broader machine learning community, and the model has been widely cited as a foundation in the chromatin deep learning literature. It catalyzed a lineage of follow-on models — including Chromoformer, which added 3D chromatin interaction data and transformer-based architectures, and TransferChrome, which added transfer learning across cell lines — that now define the state of the art. A key limitation is that AttentiveChrome operates on a fixed 10 kbp window, making it blind to distal regulatory elements and enhancer-promoter interactions beyond that range. Despite this limitation, its simplicity, interpretability, and established benchmarks continue to make it a useful reference for evaluating newer approaches.