ChromBERT (transcription regulatory networks)

Which transcription regulators occupy a given stretch of the genome, and which of them act together, changes from one cell type to the next. The experimental readout for that question is ChIP-seq, and its coverage is brutally uneven: a handful of cell lines have hundreds of profiled factors, while most cell types have almost none. That sparsity, rather than any lack of algorithms, is what has kept context-specific transcription regulatory networks (TRNs) out of reach.

ChromBERT, developed by Zhaowei Yu and colleagues at Tongji University and posted to bioRxiv in April 2025 before publication in Cell Genomics, attacks the problem by pooling the sparse data instead of working within it. Rather than reading DNA sequence, the model takes as input a binding-status vector — which cistromes are occupied at a given 1-kb genomic interval — and is pretrained with masked modeling to reconstruct the entries it cannot see. Learning to fill in a masked regulator from its neighbors forces the model to internalize the combinatorial "interaction syntax" of transcription regulators genome-wide, yielding an embedding of the local regulatory network that can then be specialized to any cellular context by lightweight fine-tuning.

The name is shared with an unrelated model: a separate ChromBERT from the University of Tokyo, catalogued here as a chromatin-state-motif model over ROADMAP annotations, is a different architecture trained on different data for a different task. This entry covers only the Tongji transcription-regulatory-network model.

#Key Features

• Occupancy as the input modality: The model ingests discretized ChIP-seq, DNase-seq, and ATAC-seq binding statuses rather than nucleotides, so the representation is about regulator co-association directly instead of inferring it from sequence.
• Prompt-enhanced cistrome imputation: Conditioning fine-tuning on a cheap, widely available assay — DNase-seq for ChromBERT-D, bulk RNA-seq for ChromBERT-R — lets the model impute regulator binding maps in cell types where no ChIP-seq for that factor exists.
• Interpretable per-regulator embeddings: Every regulator receives its own 768-dimensional vector at each genomic region, and the shift in that vector after task-specific fine-tuning ranks which regulators drive an observed regulatory effect.
• Single-cell extension: Aligning single-cell RNA-seq prompts from scGPT with the pretrained embeddings extends cistrome imputation to individual cells, where paired multi-omic binding data are essentially unavailable.
• Multiple released resolutions: Human checkpoints ship at 200 bp and 1, 2, and 4 kb resolution, with a 1 kb mouse checkpoint, alongside an installable chrombert package and documentation.

#Technical Details

Pretraining uses Cistrome-Human-6K, assembled by filtering essentially all public human ChIP-seq, DNase-seq, and ATAC-seq in the Cistrome Data Browser: 6,391 cistromes covering 991 transcription regulators, 76 histone modifications, and chromatin accessibility across more than 100 tissues. Signals are discretized into binding statuses over 1-kb intervals, giving 2,137,894 training regions; 15% of matrix entries are masked and reconstructed by a two-layer MLP decoder over a five-way status vocabulary. The encoder is an eight-block transformer with a hidden dimension of 768, deliberately matched to BERT-base. On cistrome imputation, ChromBERT-D reached an AUPRC of 0.659 against 0.514 for Avocado across the applicable test cistromes. Fine-tuned to classify causal versus non-causal eQTLs across 49 human tissues, it achieved a mean AUROC of 0.828, ahead of Enformer at 0.770, DNABERT-2 alone at 0.788, and an un-pretrained ChromBERT at 0.804.

#Applications

ChromBERT suits regulatory-genomics groups that need a TRN view of a cell type they cannot profile exhaustively. Practical uses include imputing a missing cistrome from DNase-seq or RNA-seq, prioritizing the regulators behind a differential-expression or eQTL signal, and identifying the factors that drive cell-state transitions such as differentiation or transdifferentiation without commissioning new ChIP-seq. Because fine-tuning is lightweight and the package exposes command-line tools for region, cistrome, and regulator embeddings, the model also serves as a general feature extractor for downstream regulatory models.

#Impact

By treating the regulator-occupancy matrix itself as the pretraining substrate, ChromBERT offers a complement to sequence-based genomic language models, which learn regulation indirectly through DNA. Its interpretability route — reading embedding shifts rather than attention weights — gives a concrete answer to a recurring complaint about genomic foundation models, and its comparison against Avocado, Enformer, and DNABERT-2 places it against established baselines rather than only against ablations. Limits are real: pretraining is cell-type-agnostic, so regulators and cell types are represented unevenly; the architecture's depth and hidden size were not systematically tuned; and the model covers only human and mouse. The repository ships pretrained weights via Hugging Face and Zenodo and is accompanied by a separate ChromBERT-tools package providing command-line and Python interfaces to the common workflows. Its reuse terms are unsettled: the README carries a GPLv3 badge but the repository contains no license file, the packaging metadata leaves the field commented out, and the Zenodo software deposit is labelled CC BY 4.0. The released code also covers fine-tuning only — the masked-modeling loop that produced the checkpoints is not part of it.