Reformer

An eCLIP-seq experiment tells you where an RNA-binding protein (RBP) crosslinked to the transcriptome, but it returns a peak — a region tens to hundreds of bases wide — rather than a per-nucleotide readout. That granularity is the wrong unit for a common question: whether one substitution sitting inside a peak actually perturbs binding. A peak caller cannot answer it, and the sequence models that preceded Reformer inherited the same limitation, framing RBP binding as a binary call over windows of roughly 100 bp.

Reformer, developed at Tianjin Medical University Cancer Institute and Hospital, reframes the task as regression. It is a bidirectional transformer that predicts eCLIP-seq coverage base by base across a 511 bp window from cDNA sequence alone, so the output is a binding track rather than a label. Instead of one model per protein, a single checkpoint covers every target it was trained on: the RBP and cell line enter as a prefix token prepended to the tokenized sequence (SRSF1&K562, for example), which makes conditioning a property of the input rather than of which weight file you load. It shares a name with an unrelated efficient-attention architecture from the NLP literature.

The release is a two-stage pipeline. Reformer-BC first separates binding from non-binding sites as a classifier, and Reformer then predicts affinity at single-base resolution over the regions it keeps. Both stages are initialized from pretrained DNABERT weights and fine-tuned on the eCLIP corpus. TransRBP later approached the same base-resolution problem with an added m6A signal channel.

#Key Features

• Base-resolution affinity, not a binary call: A linear head with a ReLU activation emits a predicted coverage value for each of the 509 fully tokenized positions in the input window, so peak shape and relative binding strength are both recoverable.
• One checkpoint across 155 RBPs: The target prefix token lets a single set of weights fit all 225 eCLIP-seq experiments at once, which is what makes the model reusable on an RBP and cell line the user picks at inference time.
• Context beyond the peak: The 511 bp receptive field is roughly five times the window used by earlier binary classifiers, letting flanking sequence that modulates binding contribute to the prediction.
• Attention that surfaces motifs: 872 curated RBP motifs were significantly enriched in the most-attended regions, against 486 in the eCLIP peaks themselves — 392 motifs the peak regions alone did not expose.
• Mutation scoring with wet-lab follow-through: Ranking variants by the change in predicted coverage produced candidates that electrophoretic mobility shift assays confirmed.

#Technical Details

The architecture is a BERT-style encoder: 12 transformer layers, 12 attention heads, 768 hidden units, a 512-position limit, and roughly 86 million parameters. Sequences are tokenized as overlapping 3-mers with the target prefix, [CLS] and [SEP] tokens added, and absolute position embeddings learned during training. Training data came from 225 eCLIP-seq experiments in ENCODE spanning 155 RBPs across three cell lines; peaks were expanded to a uniform 511 bp and mapped to GRCh38.p5, giving 872,618 training, 23,633 validation and 94,713 test sequences split by chromosome. Starting from DNABERT, it was fine-tuned for 30 epochs at a learning rate of 2e-5 under mean-squared-error loss on eight A100 GPUs. On the single-base-resolution test set, predicted and actual affinities correlated at a Spearman r of 0.63, with a mean Spearman r of 0.65 within individual sequences; overall binding affinity reached a mean Spearman r of 0.76 aggregated across eCLIP-seq experiments and 0.65 per individual experiment. The mean difference between predicted and measured coverage was 0.61, close to the 0.60 seen between biological replicates (paired t test, p = 0.21).

#Applications

The primary use is variant interpretation. Applied to 553,803 single-nucleotide variants drawn from ClinVar, the 1000 Genomes Project and TCGA, Reformer assigned significantly higher effect scores to pathogenic than to benign variants and to splice-site than to missense changes, concentrating high-scoring hits in genes including LDLR and LMNA. Predictions were tested directly: EMSA with recombinant PRPF8 and U2AF2 RNA-binding domains confirmed that the top-ranked substitutions altered binding in the predicted direction. A second use is motif analysis — 78 motif signatures compiled from highly attended regions reconstructed 1,038 of 1,312 ATtRACT motifs, with individual signatures tracking PUM, KH and RRM domain families.

#Impact

Reformer moved RBP binding prediction from region-level classification to a quantitative base-resolution track, which is what makes single-nucleotide effect estimates possible at all from this kind of data. Both checkpoints and the 5 GB training corpus are public. The scope is bounded by the training corpus: coverage extends only to the RBPs and cell lines represented in ENCODE eCLIP data, the window excludes longer-range regulatory context, and the model predicts eCLIP coverage — an assay-specific proxy for binding, not a thermodynamic affinity.