Prot2Token

Prot2Token is a unified framework for protein prediction that reformulates diverse biological tasks as a single autoregressive next-token prediction problem. Developed by Mahdi Pourmirzaei, Duolin Wang, Dong Xu, and collaborators at the University of Missouri, the framework was introduced in a 2024 bioRxiv preprint, extended in a March 2025 bioRxiv follow-up covering seven additional task families, and consolidated in a 2025 arXiv revision. Its central insight is that nearly every protein prediction task — whether classification, regression, binding site detection, or structure prediction — can be expressed as a sequence of tokens, allowing a single general-purpose decoder to handle them all without task-specific architectural modifications.

The approach combines established protein language model encoders, primarily ESM2, with a lightweight autoregressive transformer decoder. The decoder is conditioned on encoder embeddings and guided by learnable task tokens, allowing the model to distinguish between different prediction objectives during a single multi-task training run. Target labels are tokenized per task: class labels become discrete tokens, regression values are encoded digit-by-digit as character sequences, binding sites as sorted residue indices, and 3D coordinates as VQ-VAE structural tokens.

A key practical advantage is inference speed on structure-related tasks. By compressing structure prediction into the decoder's token generation pass, the framework runs roughly 1,000 times faster than AlphaFold2 with MSA (1–2 seconds versus 18–25 minutes for a 384-residue protein), trading accuracy against specialized methods. Across the broader task suite, Prot2Token matches or exceeds specialist models on many benchmarks while remaining a single deployable system.

#Key Features

• Unified tokenization interface: All prediction targets — including multi-class labels, real-valued regression outputs, residue-level binding sites, and 3D structural coordinates — are converted into token sequences, enabling a single autoregressive decoder to address all task types without architectural changes.
• Multi-task learning in one training run: Learnable task-prompt tokens condition the decoder on the target task at inference time, allowing joint training across heterogeneous protein prediction objectives and enabling cross-task regularization that can improve individual task performance.
• Flexible encoder compatibility: The framework is encoder-agnostic and has been tested with ESM2 at three scales (35M, 650M, and 3B parameters) and with BARTSmiles as an auxiliary chemical encoder for protein-ligand tasks.
• Self-supervised decoder pre-training: An auxiliary pre-training stage on protein sequence reconstruction improves decoder quality on spatially sensitive tasks such as binding site prediction and structure generation.
• pip-installable inference API: A PyPI package (prot2token) provides a simple interface for running inference on supported tasks without configuring training infrastructure.

#Technical Details

Prot2Token pairs a frozen or fine-tuned ESM2 protein encoder with a causal transformer decoder connected via cross-attention. The decoder configurations range from 4 layers and 8 attention heads (Prot2Token-A, paired with ESM2-35M) to 16 layers, 16 heads, and a feed-forward dimension of 5,120 (Prot2Token-D, paired with ESM2-3B). FlashAttention-2 is used throughout for memory efficiency. Training uses AdamW with cosine annealing from 1e-6 to 5e-5 and 256 warmup steps; full multi-task training in the largest configuration takes about four days on four Nvidia A100 80GB GPUs.

Across benchmark evaluations, Prot2Token achieves a Spearman correlation of 0.9294 on ProteinGym mutation stability (versus 0.613 for the prior best), a fluorescence Spearman of 0.78 with multi-task learning (versus 0.679 single-task), and an enzyme reaction classification accuracy improvement of 7.5 percentage points from multi-task learning. For localization prediction (DeepLoc 2.0), the model achieves a macro-F1 of 0.5364 versus a 0.46 baseline. Structure prediction on CAMEO 2024 yields a TM-score of 0.54, below ESMFold (0.79) but achieved roughly 1,000 times faster. The kinase phosphorylation site task reaches an F1 of 0.4966, outperforming GPS 6.0 (0.3076).

#Applications

Prot2Token suits research groups that need protein prediction across multiple task types without maintaining separate specialized pipelines. It is particularly relevant for mutation effect prediction in protein engineering, subcellular localization annotation, post-translational modification (PTM) site identification, protein-ligand and protein-protein binding site prediction, and rapid coarse-grained 3D structure estimation where throughput matters more than peak accuracy. The pip-installable package puts it within reach of bioinformaticians without deep learning infrastructure experience.

#Impact

Prot2Token contributes a conceptual shift toward generalist protein prediction systems, demonstrating that a single autoregressive decoder architecture — already dominant in natural language processing — can be adapted to the heterogeneous label spaces of computational biology. The framework's multi-task learning gains provide practical evidence that joint training across protein prediction objectives yields measurable improvements over single-task specialization. A March 2025 extension preprint from the same group broadened coverage to protein-ion binding sites, protein-protein affinity and structure similarity, substrate-kinase phosphorylation, and 3D structure prediction, and introduced the self-supervised decoder pre-training stage; the same encoder-decoder recipe underpins a separate line of work on zero-shot protein-ligand binding site prediction. A noted limitation is that sequence-to-sequence tasks such as secondary structure prediction occasionally produce outputs of incorrect length, and the authors caution that the implementation is not yet robust enough for production deployment.