Protein sequence embedding method that pools a language model's token outputs by PageRank over its own attention, adding no trained parameters.
No providers recorded yet. Browse all providers
A protein language model emits one embedding vector per residue, so proteins of different lengths leave the encoder with matrices of different sizes. Almost every downstream classifier — enzyme class, subcellular localization, interaction partner — needs a single fixed-length vector instead, and the step that gets you there is pooling. In practice that step is an afterthought: take the mean across residues, the max, the sum, or the [CLS] token. Each ignores how much residues differ in importance. Mean and sum pooling weight a catalytic histidine as heavily as a linker glycine, and the [CLS] token in a masked-language-model encoder was never trained on a sequence-level objective that would teach it to summarize.
Pool PaRTI (Pooling by PageRank Token Importance), from Russ Altman's lab at Stanford University, replaces that flat aggregation with a weighted average whose weights come from the model's own attention. It takes the attention matrices the encoder already computed, reduces them across layers, treats the result as the adjacency matrix of a directed weighted graph over residues, and runs PageRank to score how central each residue is. Those scores, normalized to sum to one, become the pooling weights.
The method introduces no learnable parameters. It never sees downstream labels and produces general-purpose embeddings rather than task-specific ones — the distinction separating it from parameterized schemes such as Light Attention and DeepLoc 2.0, whose weighting functions are optimized against one task's labels and do not transfer. Published in Bioinformatics in 2025.
The algorithm runs at inference time only. A forward pass yields token embeddings and per-layer attention; pixel-wise max pooling across layers collapses attention into one matrix, capturing the strongest relationship at any depth rather than the last layer alone. That matrix defines a fully connected directed graph over residues, on which PageRank runs with damping factor 0.85, a cap of 100 iterations, and tolerance 1e-6. Normalized node scores weight the average of token embeddings. Cost is quadratic in sequence length, measured at polynomial degree 2.068 on CPU.
Evaluation spans four tasks against [CLS], mean, max, and sum pooling. With ESM-2, Pool PaRTI leads every baseline on all four: SCOP fold retrieval precision@10% of 0.508 and MRR 0.682; enzyme class retrieval 0.429 and 0.710; protein-protein interaction accuracy 0.646, MCC 0.291, AUPRC 0.699; and subcellular localization accuracy 0.905 with MCC 0.572 and Jaccard index 0.639, against 0.873, 0.306, and 0.520 for mean pooling. With ProtBERT the advantage narrows to two of the four tasks — sum and mean pooling edge it out on interaction accuracy and enzyme retrieval precision — which the authors attribute to ProtBERT's 420M parameters producing lower-resolution attention than ESM-2's 650M. Against Light Attention, which needs 29.5M extra trained parameters, it wins on enzyme prediction at full and half data and ties at a quarter.
Pool PaRTI is a drop-in replacement wherever a protein-level vector is currently produced by mean pooling: enzyme function annotation, subcellular localization, interaction prediction, proteome similarity search, and feature generation for small-data models where a parameterized pooling head would overfit. Needing no labels, it suits exploratory and data-scarce work. The residue importance scores are useful independently of the embedding, highlighting candidate functional sites from sequence alone and prioritizing mutagenesis positions in proteins with no solved structure.
Pool PaRTI shows that a step most pipelines treat as plumbing carries real signal, and that the information needed to weight it well already sits unused inside the encoder. Framing attention as a graph and importing PageRank costs one inference pass and no training, lowering the barrier to adoption compared with learned pooling heads. The limits are stated in the paper: gains depend on attention quality and shrink with a weaker backbone; validation covers encoder-only models and does not transfer as-is to autoregressive decoders or attention-free state space models; and normalizing weights makes embedding magnitude length-independent, a poor fit for properties that scale with sequence length, where sum pooling retains an edge. The code is released under Apache-2.0.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.