ProtAC

A protein language model trained to predict function learns whatever its labels say, and most of the labels in a database the size of UniRef50 were never read by a human. Swiss-Prot's manually reviewed entries are the curated core; the tens of millions of records around them carry Gene Ontology terms propagated by automatic pipelines, some wrong, some stale, some too coarse for the sequence they sit on. Training longer on that corpus does not fix it, and no held-out split tells the model which of its training labels a curator would have rejected.

ProtAC — Protein Automatic Cleaning — makes the model an auditor of its own training set. Developed at the Hong Kong University of Science and Technology (Guangzhou) with Shanghai Jiao Tong University and posted to bioRxiv in October 2024, it runs a three-stage cycle: pretrain on the noisy corpus, fine-tune on the manually curated one, then use the fine-tuned model to re-annotate the noisy corpus. A Sequence-Annotation Matching (SAM) filter adjudicates each proposed replacement, and the cleaned corpus becomes the next round's pretraining set. Corpus quality and model quality improve together across rounds instead of one being frozen before the other — an idea borrowed from the caption filtering used in vision-language pretraining.

The name collides with PROTAC in the small-molecule sense — proteolysis-targeting chimeras — but the two are unrelated; this acronym expands to Protein Automatic Cleaning and the model touches no degrader chemistry. The same group later released AnnoDPO, a separate model attacking annotation through preference optimization rather than recursive cleaning.

#Key Features

• Sequence-annotation matching filter: A two-class linear head reads the fused sequence-annotation embedding and scores whether the pair belongs together. A model-proposed annotation replaces the database's own only when it both matches the sequence and matches it more strongly than the original.
• Recursive cleaning cycles: Each round is pretrain, fine-tune, then caption; the cleaned corpus supplants the noisy one in the following round. SAM accuracy, GO F-max, AUROC and recall all rise monotonically across four rounds.
• Annotations as a trained modality: Functional labels enter as a fixed-size binary vector over 7,533 GO terms and pass through their own encoder, with cross-attention layers injecting sequence context — not a classification head stacked on frozen embeddings.
• Checkpoints at five sizes: ProtAC-ESM2-8M, -35M, -150M and -650M plus ProtAC-PB span 27M to 824M total parameters, all loaded through the same evaluation entrypoint.
• A corpus, not only a model: The re-annotated UniRef50 snapshot is released alongside the weights, usable as pretraining data for unrelated protein models.

#Technical Details

Four modules make up the network. The sequence encoder is the local branch of ESM-2 (8M, 35M, 150M or 650M) or of ProteinBERT; the annotation encoder is a modified ProteinBERT global branch with an added cross-attention layer per block; the annotation decoder repeats that structure over fully masked inputs to predict the label set; and the SAM filter is a single linear layer. Three losses are optimized jointly: masked language modeling over 15% of sequence positions, sequence-annotation matching with ALBEF-style in-batch hard negatives (focal loss in practice, since negatives outnumber positives two to one), and binary cross-entropy annotation prediction.

Pretraining combines UniRef50 as of May 2018 (30.16 million sequences, the noisy set) with the Swiss-Prot training split; Swiss-Prot as of July 2023 supplies 560,000 curated entries, roughly 530,000 for fine-tuning and 30,000 held out. Annotation dictionaries cover 7,533 GO terms and 753 keywords, and training ran on eight A800 GPUs at batch size 256 with AdamW. On the DeepFRI benchmark splits at a 95% sequence-identity threshold — the task configuration used by ProtST — ProtAC-ESM2-35M reaches F-max 0.379 on GO-BP, 0.603 on GO-MF, 0.461 on GO-CC and 0.619 on EC, the best GO-CC among models under 100M parameters and close behind the far larger OntoProtein on GO-BP and GO-MF. Both ESM-2-based variants beat their own pretrained backbones on all four tasks, and on keyword prediction the cleaned-corpus ProtAC-ESM2-35M reaches AUROC 0.8602 and F-max 0.6802. Pretraining from scratch on the four-round cleaned corpus lifted peak F-max 13.9% over the original corpus and reached the original's best score in 38.9% less training time.

#Applications

The direct use is functional annotation of sequences with no close curated homolog: GO term and UniProt keyword prediction from sequence alone, with the SAM score attached to each proposal so a curator sees a ranked, auditable list rather than a flat label dump. Manual review of sampled UniRef clusters found ProtAC's added GO terms consistent with InterPro family and domain evidence and with later UniProt revisions, and its removals concentrated on terms the databases themselves dropped. Beyond annotation, the cleaned UniRef50 release is drop-in pretraining data for other protein models, and the encoder pairing is deliberately backbone-agnostic — the sequence and annotation branches can be swapped for other pretrained protein language models.

#Impact

ProtAC's contribution is less a leaderboard position than a demonstration that a protein corpus and the model trained on it can be improved in the same loop, with an explicit, inspectable decision rule for every label change. Its practical caveats are worth stating plainly. The preprint's benchmark tables cover only the sub-100M variants — ProtAC-PB, ESM2-8M and ESM2-35M — while the 150M and 650M checkpoints in the weight release carry no published numbers. Ablations show the matching and masked-language losses are not uniformly helpful: dropping either raises GO F-max during pretraining while degrading the captioning and EC tasks the cleaning cycle depends on. The work remains a preprint awaiting peer review, and the repository describes itself as under construction. Code and weights are MIT-licensed; the preprint carries a CC BY-NC-ND 4.0 license.