DNABERT-Epi

CRISPR/Cas9 genome editing cuts where the single-guide RNA directs it, but it also cuts at sites that merely resemble the guide. Predicting those off-target cleavages is a prerequisite for therapeutic editing, and the deep learning models built for the task have almost all been trained from scratch on off-target data alone — a few thousand labelled positive sites against millions of negatives, with no access to the broader statistics of the genome itself.

DNABERT-Epi, developed by Kai Kimata and Kenji Satou at Kanazawa University, attacks that data bottleneck from two directions. It starts from DNABERT, a BERT-style DNA language model pretrained on the human genome, fine-tunes it to score guide-target pairs, and then fuses in the chromatin context of each candidate site through a gated multimodal head. The premise is that genome-scale pretraining supplies sequence priors that off-target datasets are too small and too imbalanced to teach, while epigenetic tracks supply the cellular context that explains why an in vitro-detectable site does or does not get cut inside a living cell.

The work was posted as a preprint in April 2025 and published in PLOS ONE that November. The authors position it as the first demonstration that a pretrained DNA foundation model confers a measurable advantage on CRISPR/Cas9 off-target prediction, supported by ablations that separate the contribution of pretraining from that of the epigenetic features.

#Key Features

• Two-stage fine-tuning: The backbone is first fine-tuned on a mismatch-position prediction task that teaches it the pairing relationship between sgRNA and target DNA, then fine-tuned again for binary off-target classification.
• Bulge-aware tokenization: The 3-mer vocabulary is extended with tokens containing a gap character, so insertions and deletions in the guide-target duplex are represented natively rather than dropped.
• Gated epigenetic fusion: A gate vector derived from the sequence embedding modulates the epigenetic embedding element-wise, letting the model down-weight chromatin signal when the sequence context alone is decisive.
• Seven-dataset benchmark: Evaluation spans one in vitro and six in cellula datasets, with sgRNA-grouped cross-validation on three and four held out entirely as independent test sets spanning different cell types and detection assays.
• Attribution analysis: SHAP, integrated gradients, and attention-weight visualization identify which epigenetic marks and sequence positions drive each prediction.

#Technical Details

The sequence backbone is the 3-mer DNABERT model — a 12-layer, 12-head transformer encoder — loaded from its public checkpoint and fine-tuned with inputs formatted as [CLS] sgRNA 3-mers [SEP] DNA 3-mers [SEP]. Both fine-tuning stages use a learning rate of 2e-5 over five epochs. For each candidate site, H3K4me3, H3K27ac, and ATAC-seq signal is extracted from a 1,000 bp window centred on the cleavage site, outlier-capped, Z-scored, and binned into 100 bins of 10 bp per mark, giving a 300-dimensional epigenetic vector that an MLP embeds before gating and concatenation with the [CLS] embedding. Training data comes from the CHANGE-seq in vitro set (110 sgRNAs, 202,041 active and 4,936,279 inactive sites), the Lazzarotto GUIDE-seq set (78 sgRNAs, 2,166 active sites), and the Schmid-Burgk TTISS set (59 sgRNAs), with negatives downsampled to 20% during training only.

On the GUIDE-seq benchmark, DNABERT-Epi reached a PR-AUC of 0.550, the best of any single model, ahead of the sequence-only fine-tuned DNABERT at 0.539 and CrisprBERT at 0.511, and it improved significantly over sequence-only DNABERT on F1, MCC, and PR-AUC while showing no ROC-AUC gain. Ablating pretraining was far more costly than ablating epigenetics: random initialization cost 0.1653 PR-AUC, whereas removing the epigenetic branch cost 0.0115. A soft-voting ensemble over all seven models — the five reimplemented baselines plus DNABERT and DNABERT-Epi — outperformed every individual model on every dataset.

#Applications

The intended use is triaging candidate off-target sites during guide design for therapeutic and functional-genomics editing, where missing a cleavage event in a coding or regulatory region is costly and validating every genome-wide candidate is infeasible. Because the epigenetic branch takes cell-type-specific ATAC-seq and histone ChIP tracks, predictions can be conditioned on the chromatin state of the intended target cell rather than treating the genome as naked DNA — useful when the same guide behaves differently in primary T cells than in an immortalized line.

#Impact

The clearest result is the ablation: genomic pretraining, not the epigenetic features, supplies most of the gain, which argues for building CRISPR predictors on DNA foundation models rather than training bespoke architectures on scarce off-target labels. The epigenetic contribution is statistically significant but small, and it comes at the cost of requiring matched chromatin tracks for the target cell type, which limits the multimodal variant to the one dataset with suitable data. The released repository provides PyTorch reimplementations of all five baselines alongside the DNABERT-Epi training code, making it a usable benchmark harness; no trained checkpoint is distributed, so applying the model to new guides means rerunning the fine-tuning pipeline. The repository carries no license file.