Loading…
Protein language model for microbial smORF-encoded small proteins, pairing multi-scale convolutions with transformer layers in a compact encoder.
No providers recorded yet. Browse all providers
An anti-CRISPR protein is often under 100 residues, and what separates it from an ordinary phage peptide is a short run of adjacent positions rather than a fold or a conserved domain. Protein language models trained on UniRef50 and UniRef90 have learned the opposite prior — long-range attention, global structural dependency, evolutionary alignment depth — and on the PaCRISPR anti-CRISPR benchmark all three pretrained baselines tested here collapsed to majority-class prediction, scoring an F1 of 0.000 and an AUC at or below chance. TinyProteinTransformer (TPT) is built around that mismatch, with inductive biases chosen for short, motif-driven, rapidly evolving sequences instead of long conserved ones.
The architecture puts four parallel 1D convolutions, with kernel sizes of 3, 5, 7 and 9 residues spanning typical short-protein motif lengths, in front of a 20-layer transformer encoder, then condenses residues into a single vector with a learned attention-pooling operator rather than a mean. Pretraining is self-supervised on the Global Microbial smORF Catalog and combines masked language modeling with a down-weighted InfoNCE contrastive term whose two views of each peptide come from short random deletions and chemically conservative substitutions — augmentations chosen to imitate the truncations and neutral mutations that short microbial proteins actually undergo.
TPT was developed by Fang Sheng, Junhe Zhang and Chengkai Zhu at the Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, with collaborators at the University of Toronto and Zhongshan Hospital, and published in Frontiers in Microbiology. Training code is released under MIT and the checkpoints under Apache 2.0.
TPT holds 103M parameters across a 640-dimensional hidden state, 20 pre-norm transformer layers, and convolutional branches of 256 channels each, with a hard input ceiling of 128 residues. Pretraining ran for three epochs over GMSC10.90 — 287,926,875 non-redundant smORF families clustered at 90% identity, assembled from 63,410 metagenomes and 87,920 isolate genomes — using AdamW at a learning rate of 2e-5 in mixed precision on two RTX 5090 GPUs. A hidden-dimension sweep over 320, 480, 640 and 800 peaked at 480 and 640, and 640 was adopted as the default; the 800-dimensional variant lost ground, consistent with overfitting a rare minority class.
Evaluation is frozen-encoder linear probing under 5-fold stratified cross-validation on six third-party benchmarks: antimicrobial peptides, toxic peptides, bacteriocins, anti-CRISPR proteins, quorum-sensing peptides and cell-penetrating peptides. TPT reached an AUC of 0.930 on antimicrobial peptides against 0.928 for ESM-2 at 150M parameters, and 0.930 versus 0.925 on toxicity — differences the authors report as not statistically significant. The distinguishing result is anti-CRISPR, where TPT scored 0.696 against 0.428 for ESM-2 150M and 0.322 for ProtBERT from the ProtTrans suite, and was the only encoder with a positive silhouette coefficient. Inference took about 6.6 ms per sequence versus 28.6 ms for ESM-2 150M. Ablating the contrastive objective cost more than removing the CNN branch or attention pooling, dropping antimicrobial-peptide AUC to 0.869.
The immediate use is screening candidate small proteins mined from metagenomes: an embedding plus a linear probe ranks smORF products for antimicrobial activity, toxicity, bacteriocin identity, anti-CRISPR function, quorum sensing or cell penetration — the filtering step ahead of synthesis and assay in microbiome-derived peptide discovery. Because a forward pass costs single-digit milliseconds and the checkpoint is a few hundred megabytes, screening millions of candidate smORFs is tractable on one GPU.
TPT is a concrete argument that domain-matched inductive bias can substitute for scale on short sequences, and the anti-CRISPR result gives it teeth: general protein language models did not merely underperform there, they failed outright. The evidence has real bounds. The cross-model comparison uses linear probing on frozen embeddings, which the authors note may understate what end-to-end fine-tuning achieves; several benchmarks are small and unbalanced, with bacteriocins comprising 242 sequences and anti-CRISPR 294; and near-identity overlap between GMSC and those small benchmarks varies by dataset, which the authors quantify with a shipped MMseqs2 script. The model carries no structural supervision and cannot accept sequences beyond 128 residues, so it is a specialist encoder for small proteins, not a general-purpose replacement.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.