Loading…
Predicts protein properties from sequence alone by LoRA fine-tuning ESM-2 and ESM-C backbones, with contact maps biasing attention pooling.
No providers recorded yet. Browse all providers
A lab holding a few thousand labelled proteins and a single GPU faces an awkward choice. Freezing a protein language model and training a small head on top is cheap, but the frozen representations were never shaped by the task. Fully fine-tuning a backbone the size of ESM-2-650M costs memory the lab does not have and overfits readily. SeqProFT answers systematically: adapt the backbone with low-rank updates, then spend the rest of the budget on the pooling step that turns per-residue embeddings into a prediction.
Developed by Shuo Zhang and Jian K. Liu at the School of Computer Science, University of Birmingham, the framework injects Low-Rank Adaptation (LoRA) matrices into the projection weights of a frozen backbone and pairs them with one of three downstream heads. The most distinctive head uses a residue-residue contact map — read out of the backbone's own attention scores rather than from a structure — as a multiplicative bias on the head's attention weights, so aggregation favours residue pairs likely to be in contact. The input stays strictly sequence: no alignments, no coordinates, no folding step.
The peer-reviewed version spans two backbone families: ESM-2 at 35M, 150M and 650M parameters, and ESM Cambrian (ESM-C) at 300M and 600M. Its central finding is that adaptation substitutes for scale — ESM-2-150M with LoRA beats ESM-2-650M without it on enzyme commission classification (0.854 against 0.838) and GB1 fitness (0.957 against 0.881), a 75.3% cut in parameters for better results.
Low-rank adaptation of frozen weights: LoRA matrices with rank 32 and scaling factor 32 sit in the query, key, value and dense projections. Training touches 35.56M of ESM-2-650M's 685.80M parameters (5.18%), rising to 15.05% for the 35M backbone.
Three interchangeable prediction heads: a Simple MLP Head (SMH) with learned attention pooling, a Multi-head Attention Head (MAH) running four-head self-attention over the residue embeddings, and a Contact Map enhanced MAH (CM-MAH), matched to task complexity.
Contact maps as an attention bias: CM-MAH unfreezes the backbone's contact head and multiplies the head's attention logits by the learned contact probabilities, so structural signal enters from co-evolutionary information latent in the pretrained attention.
One recipe across ESM generations: the same adapters and heads run over ESM-2 and ESM-C unchanged, and the gains hold on both. The largest single improvement from LoRA is +27.84% on fold-superfamily classification with ESM-C-600M.
Adaptation sharpens attention: per-layer attention entropy falls after LoRA, most steeply in layers 12–20, concentrating attention on residues that are known functional sites.
Every experiment fits on a single NVIDIA A100 40 GB GPU. Evaluation covers ten tasks over twelve subsets — EC numbers, the three GO branches, fold classification at family, fold and superfamily level, secondary structure, localization, the fluorescence landscape, HumanCell thermostability and GB1 fitness — on splits from GearNet, Ankh and FLIP.
With ESM-2-650M and CM-MAH, SeqProFT reaches 0.886 F1-max on EC, 0.674 on GO molecular function, 0.746 accuracy on fold-superfamily, 0.825 on secondary structure, 0.831 on localization and Spearman 0.954 on GB1; ESM-C-600M edges past it on several, including 0.888 on EC. Contact enhancement helps classification most — +8.3% on fold-fold and +6.7% on fold-superfamily over SMH — and is close to neutral on regression. A rank sweep over 1, 2, 4, 8, 16 and 32 leaves performance largely flat, so ranks of 1–8 recover most of the benefit.
SeqProFT is a fine-tuning recipe rather than a shipped predictor, for groups with neither structures nor a large GPU budget. Functional annotation is the immediate use: EC number, GO term and localization models trained this way can run over a newly assembled proteome. Protein engineers get fitness prediction on epistatic landscapes such as GB1 and the GFP fluorescence set for prioritising library designs, and thermostability regression for developability triage. The repository ships per-task configuration files, so retargeting means swapping data and a config.
The contribution is a controlled measurement of what parameter-efficient adaptation buys across a broad task panel, and the answer — a smaller adapted backbone routinely matches or beats a larger unadapted one — is practical guidance for groups without cluster access. Against sequence-only baselines SeqProFT leads on most shared tasks, and on GO cellular component (0.534) it exceeds every compared structure-based method. Extending the study from ESM-2 to ESM-C shows the recipe is not tied to one backbone generation, which matters more than any individual number as these models turn over.
Three limitations bound the results. No fine-tuned checkpoints are released and the repository carries no license, so the metrics can be reproduced only by retraining. Each head is trained independently per task, with no transfer between tasks. And for secondary structure and localization the splits supply no validation set, so training length was selected on the test set; the authors flag this and frame their claims as relative gains.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.