Loading…
Protein language model reading each residue alongside an unsupervised local-fragment token, so one encoder serves residue- and chain-level tasks.
No providers recorded yet. Browse all providers
A protein language model that tokenizes one residue at a time works with a vocabulary of
about twenty symbols, each far less distinctive than an English word — and masked language
modelling gets easier the more distinctive its vocabulary is. The obvious fix is to merge
recurring runs of residues into subword tokens, the way PRoBERTa and BPE
Longformer do. That fix costs something specific: once QRI is one token, there is no longer
any representation to hang a per-residue label on, so a subword model can classify a fold or
a family and nothing finer — secondary structure and residue contacts drop out of reach.
SPRoBERTa keeps both units at once. Every input position carries an amino acid embedding and the embedding of the local fragment that residue falls inside, each with its own position embedding, summed before they reach the encoder. The fragments come from a SentencePiece unigram model fitted to the raw FASTA corpus — a tokenizer learned from the data rather than fixed k-mers — and a second pretraining objective masks whole fragments and asks the model to recover them, running alongside ordinary per-residue masked language modelling. The result is one pretrained encoder that fine-tunes to residue-level, residue-pair-level and whole-chain tasks alike.
It was built by Lijun Wu and colleagues at Microsoft Research Asia, with Nanjing University and the University of Science and Technology of China, and published in Briefings in Bioinformatics in September 2022. One letter apart in name, it is not PRoBERTa (Nambiar et al.), sharing no authors, tokenizer or task set with it.
The encoder follows the RoBERTa base configuration: 12 transformer layers, 768-dimensional embeddings, 3072 feed-forward units and 12 attention heads, trained for roughly 200k steps with Adam at a 0.0006 peak learning rate, batches of 4096 tokens and a 768-residue length cap. Fragment vocabularies of 10k, 30k and 50k were trained separately. Pretraining ran on TAPE's processed Pfam split of 31 million protein domains, and again on UniRef50.
On CB513 secondary structure, SPRoBERTa reaches 0.769 accuracy from Pfam against 0.752 for the same RoBERTa backbone trained on residues alone and 0.730 for TAPE; UniRef50 pretraining lifts it to 0.818, past the 0.800 MSA alignment baseline. Contact prediction on TAPE's CASP12 set (L/5 precision, medium and long range) goes from 0.376 to 0.516 at a 30k vocabulary, and to 0.632 from UniRef50 — well short of the 0.569 long-range figure ESM-1b posts on the ESM test set with stacked convolutional fine-tuning heads, where SPRoBERTa reaches 0.395. Remote homology accuracy is 0.304 against 0.256 for BPE Longformer. On GO term prediction it records 0.623 AUPRC for cellular component, above ESM-1b's 0.384, and 0.604 for molecular function against 0.639. Removing the fragment-recovery objective drops 30k contact precision from 0.516 to 0.416, the single largest ablation effect reported.
The model is used by fine-tuning the pretrained encoder with a task head, which suits groups annotating secondary structure, predicting residue contacts as a structure prediction input, assigning folds by remote homology, or attaching GO terms to unannotated sequences. Its more portable contribution is the recipe: the fragment track is an additive change to any residue-level encoder, and the tokenizer-size sweep gives a concrete read on how much vocabulary a protein corpus supports.
SPRoBERTa argued that tokenization granularity is a lever on protein representation quality independent of parameter count — a 12-layer model closing much of the gap to far larger encoders on structural tasks. No code or trained weights accompany it: the article carries no code or data availability statement and no repository or checkpoint has surfaced, so the numbers cannot be reproduced from released artifacts and the model is not usable off the shelf. Uptake has been correspondingly modest, though the idea persisted inside the same group — Microsoft's later SFM-Protein retains a span-prediction head over subword tokens beside residue-level masking. The authors name two limits: two embedding matrices inflate model size, and because fragments are added rather than substituted, sequence length is never reduced, leaving long chains as costly as before.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.