VenusPLM

Protein lengths span two orders of magnitude — a 30-residue peptide and a 3,000-residue titin fragment land in the same training batch — so a language model that pads every sequence to the batch maximum spends most of its attention compute on padding tokens. VenusPLM is built around the alternative: many sequences are concatenated end to end into one long context, and the attention mask carries a per-sequence index that keeps each protein from attending across its neighbours' boundaries. The packed batch does the same work with far fewer wasted positions, and FlashAttention keeps the long concatenated context cheap.

That efficiency is what let the model be trained on VenusPod, a corpus of over nine billion protein sequences assembled from UniRef100, the Observed Antibody Space, NCBI, GOPC, MGnify and JGI. The mix matters as much as the size: alongside the curated UniRef backbone it carries antibody repertoires and metagenome-assembled sequences from environmental sampling, regions of sequence space that a UniRef-only corpus represents thinly. Pretraining is a corruption-recovery objective — the model predicts residues that have been removed and residues that have been swapped for a different amino acid, so it learns to recognise a substitution as wrong rather than only to fill a blank.

VenusPLM was released in 2025 by the ai4protein group, supported by Liang Hong's laboratory at Shanghai Jiao Tong University, the same group behind ProSST and ProtSSN. Where those add an explicit structural channel, VenusPLM stays sequence-only and puts its budget into corpus breadth and throughput. No paper accompanies the release; the architecture and corpus details above come from the project's own repository, and its evaluation from third-party benchmarking.

#Key Features

• Sequence packing with boundary-aware attention: Multiple proteins share one context window, separated by an index in the attention mask rather than by padding, which raises inference and training throughput on the short sequences that dominate real corpora.
• Substitution-aware pretraining: The objective covers both deleted and substituted residues, so the model is trained to score how plausible a specific amino acid is at a position, not merely to fill a masked slot.
• Single-pass mutational scanning: One forward pass over a wild-type sequence yields log-probabilities for all 20 amino acids at every position; the log-odds of mutant against wild type ranks the complete single-site landscape with no assay data.
• Drop-in embeddings: A pip install vplm package exposes the checkpoint through a HuggingFace-style interface, returning per-residue hidden states for downstream property predictors without any fine-tuning.

#Technical Details

VenusPLM-300M is an encoder-only transformer of 302,114,816 parameters: 24 layers, hidden size 1024, 16 attention heads, a 4096-dimensional feed-forward block with SiLU activations, RMSNorm, and rotary position embeddings with base 10,000, over a 36-token vocabulary. The rotary scheme means there is no learned position table to cap sequence length. Weights ship as a single 1.2 GB float32 safetensors file, and the package supports naive, SDPA and FlashAttention backends, the last in bfloat16.

PFMBench, an independent benchmark of 17 protein foundation models across 38 tasks from BioMap Research and Westlake University, evaluated the checkpoint without retraining. On zero-shot ProteinGym it scores Spearman 0.43952, ranking second among the nine models the protocol could evaluate — ahead of ESM-2 650M at 0.43904 and behind SaProt 650M at 0.45094, both more than twice its size. Under adapter tuning it reaches 0.7519 on enzyme commission classification against ESM-2's 0.7358, and beats ESM-2 on half of the eight representative supervised tasks in that comparison.

#Applications

The two shipped workflows cover most of what a protein engineering group needs from a sequence model. Scoring the full single-site mutational landscape of an enzyme or binder gives a ranked shortlist to order and assay, the standard first pass before a directed evolution campaign. Extracting frozen per-residue or pooled embeddings feeds downstream classifiers and regressors for solubility, stability, localisation and function annotation — the adapter-tuning setting PFMBench measures. Because the interface mirrors HuggingFace conventions, either use drops into a pipeline already built around another protein language model.

#Impact

VenusPLM's interest lies in the trade it makes: a 300M-parameter sequence-only model that matches or edges past 650M sequence models on zero-shot fitness prediction, which points at corpus composition and training throughput rather than parameter count as the lever worth pulling. Its release is unusually thin on documentation — there is no paper, no Methods section, no published training recipe, and VenusPod has not been released, so the corpus cannot be independently checked and the training run cannot be reproduced. Licensing is restrictive and its record is stale in places. The project shipped under MIT in March 2025 and was relicensed to CC-BY-NC-ND-4.0 on 2025-05-16, barring commercial use and distribution of adapted material — but the vplm package on PyPI was uploaded before that change and still advertises MIT, since its setup.py reads the LICENSE file at build time. The HuggingFace repository carries no license at all, so the weights are the one artifact whose terms nobody states. Anyone planning to fine-tune the checkpoint or use it commercially should resolve that with the authors.