DrugPLMFormer

Deciding whether a protein is worth a screening campaign usually waits on a structure: a pocket has to be found, measured, and judged ligandable. For most of the proteome that evidence is missing, of uncertain quality, or unlinked to any curated druggability label, which leaves early target triage with nothing to rank on. DrugPLMFormer treats druggability as a sequence classification problem instead, and pushes the representation work onto a protein language model that has already absorbed structure-relevant regularities from raw sequence during pretraining.

Built at the Department of Biomedical Engineering at Meybod University in Iran, the model keeps ESM-2 frozen and precomputes its residue-level token embeddings once. Those cached embeddings feed a downstream stack that reads the sequence at two scales simultaneously: a self-attentive BiLSTM for local and mid-range ordering, and a Transformer encoder for long-range dependencies across the chain. A gated fusion layer optionally folds in compact physicochemical descriptors — reduced-alphabet groupings by charge, polarity, aromaticity, secondary-structure propensity and residue size — before an MLP head emits a single druggable/non-druggable probability. Nothing in the backbone is updated: a new sequence goes embedding → fixed head → score.

The second half of the contribution is procedural rather than architectural: informal tuning is replaced by a compute-budgeted search, and the splits are built to make homology leakage visible, so the reported numbers describe out-of-cluster behavior rather than memorized neighbors.

#Key Features

• Frozen, cached representations: ESM-2 parameters are never updated and token embeddings are computed once and reused, which confines task-specific adaptation to a small downstream classifier and makes repeated experiments cheap to rerun.
• Two-scale sequence encoder: The self-attentive BiLSTM captures ordered local and mid-range patterns while the Transformer branch adds global attention; ablations show both contribute, with the Transformer mattering most for ranking quality.
• Budget-aware model selection: Architecture, training and operating-point hyperparameters are chosen by a Bayesian-optimization/CTCM hybrid under a fixed evaluation budget, using validation AUPRC and MCC as the selection objective.
• Homology-aware evaluation: The development corpus is clustered with MMseqs2 at 50% sequence identity and whole clusters are assigned to a partition, with a conventional stratified split kept only as a secondary reference.
• Optional descriptor fusion: Physicochemical features enter through a gated layer as complementary evidence rather than as the primary input, keeping the learned representation in charge.

#Technical Details

ESM-2 was selected as the backbone over ProtBERT and ProtT5 under a matched frozen-embedding protocol, on validation AUPRC and MCC. The downstream network carries 5.72M parameters, 1.59G multiply-accumulates at a post-processed length of 256 residues, and a 22.9 MB FP32 footprint — the count excludes the frozen backbone. Development used ProTar-II, 4,068 sequence-only proteins balanced 2,034 druggable to 2,034 non-druggable, split roughly 70/15/15 by MMseqs2 cluster. Training used AdamW at a 1e-4 learning rate, batch size 64, dropout 0.30, up to 50 epochs and 10 random seeds. On the homology-aware internal test partition the model reached 95.98% accuracy, 96.01% F1, 96.42% sensitivity, 95.61% specificity and 0.981 ROC-AUC. Two corpora were held out entirely — never used for fitting, early stopping, threshold selection or hyperparameter search: ProTar-II-Ind (450 proteins) at 96.62% accuracy and 0.9688 ROC-AUC, and DPI_CDF (2,542 proteins, DrugBank positives against Swiss-Prot negatives) at 96.20% accuracy and 0.9696 ROC-AUC. Restricting to external proteins under 50% identity to the training partition — 65.3% and 69.6% of each set — performance held at 96.18%/0.966 and 95.91%/0.968. With embeddings already cached, downstream CPU scoring takes roughly 1.02–1.22 ms per sequence.

#Applications

The intended use is early target triage: ranking candidate proteins before committing structural, biochemical or screening resources, particularly for targets whose structures are unsolved, low-confidence, or unlinked to druggability annotations. Because the backbone is frozen and its embeddings are reusable, a proteome can be embedded once and rescored repeatedly at negligible marginal cost, which suits proteome-wide sweeps and periodic re-prioritization as target libraries change.

#Impact

DrugPLMFormer is a careful piece of evaluation engineering more than a new representation, and the authors bound their own claims accordingly. They frame the work as retrospective screening and note that paired external comparisons found accuracy and F1 differences numerically favorable but not statistically significant; only the ROC-AUC improvement on DPI_CDF was statistically supported (Δ = +0.0138, p = 0.0036). DPI_CDF negatives are operational labels — proteins not annotated as drug targets — not experimentally demonstrated non-druggability, so cross-dataset agreement carries label heterogeneity with it. Frozen embeddings also inherit whatever the pretraining corpus contained. No code, weights or hosted tool have been released; the article's data availability statement promises reproducibility materials in future tense without naming a host, and the article itself is open access under CC BY-NC-ND 4.0. What the work does establish is that a small classifier over frozen protein-language-model embeddings, selected and tested under homology control, transfers to disjoint druggability corpora without refitting.