DSM (Diffusion Sequence Model)

Protein language models have split into two camps that rarely overlap. Masked language models such as ESM-2 produce excellent residue-level representations for downstream prediction but generate poorly, because filling in a handful of masked positions is a different problem from writing a sequence from nothing. Autoregressive models generate fluently but their left-to-right representations tend to lag on transfer tasks. Practitioners end up running two models.

The Diffusion Sequence Model (DSM), from the Gleghorn Lab at the University of Delaware and Synthyra, collapses the two into one training scheme. It takes the ESM2 architecture and replaces masked language modeling with a masked diffusion objective adapted from the LLaDA framework: rather than masking a fixed 15% of tokens, the corruption rate is sampled across the full range up to complete masking, and the model learns to reconstruct the entire sequence in a single forward pass at every noise level. Training at high corruption teaches genuine generation; training at low corruption preserves the representational quality that makes masked models useful.

The result is a single checkpoint that both embeds and designs. DSM produces biomimetic sequences whose amino-acid composition, predicted secondary structure, and predicted function track natural proteins even when 90% of the input is masked, while its embeddings match or beat comparably sized pLMs on downstream probes. A fine-tuned variant, DSM-ppi, conditions generation on a target sequence to design binders.

#Key Features

• Unified representation and generation: One masked-diffusion objective yields both transferable embeddings and a usable generative model, removing the need to pair a masked encoder with a separate design model.
• Robust to extreme corruption: At a 90% mask rate DSM150 reaches an alignment score of 0.277 against roughly 0.15 for ESM2-650, and generated sequences hold a Jensen-Shannon divergence below 0.01 against natural amino-acid k-mer distributions.
• Metagenomic-scale pretraining: Training runs on OMGprot50, over 207 million sequences obtained by clustering the Open MetaGenomic dataset at 50% identity, an evolutionarily broader corpus than UniRef-based alternatives.
• Target-conditioned binder design: DSM-ppi attends to a target sequence while generating its partner, supporting template-guided screens in which known binders are randomly masked and rewritten.
• Open weights across scales: 150M and 650M base checkpoints plus LoRA and full-finetune DSM-ppi variants are published on HuggingFace with training and evaluation code.

#Technical Details

DSM extends pretrained ESM2 checkpoints rather than training from scratch. The language-modeling head gains an extra linear transformation and a soft-logit cap, and the output projection is tied to the token embedding matrix. DSM150 was trained for 100,000 steps at batch size 32 and sequence length 512; DSM650 used the same schedule with global batch size 128 and length 2048, costing roughly 12 days on four A100 GPUs. The training loss follows LLaDA's masked-diffusion formulation with uniformly sampled mask rates, and all masked tokens are predicted in one pass rather than iteratively.

On representation probes spanning secondary-structure prediction and other supervised tasks, DSM650 produced the strongest embeddings among similarly sized pLMs benchmarked against ESM2, ESM-C, ProtBert, Ankh, DPLM, and gLM2, surpassed on average only by the much larger ProtT5. For binder design the authors screened 1.4 million candidates — 100,000 per target and template scheme — across the seven BenchBB targets (EGFR, IL-7Ra, MBP, PD-L1, BBF-14, BHRF1, Cas9), with top designs exceeding the predicted binding affinity of the best publicly known binders; DSM650-ppi recorded the lowest cross-entropy of any model tested at 1.989. The project repository additionally reports biolayer-interferometry testing of 40 designs, in which 12 of 13 expressed EGFR variants bound and the best reached 861 pM affinity. Code and weights carry a custom Gleghorn Lab/Synthyra license that permits academic and commercial use subject to a biosecurity acceptable-use clause.

#Applications

DSM suits groups that want one model for both embedding and design: featurizing sequences for property prediction, filling in or diversifying regions of an existing protein, and generating de novo candidates. DSM-ppi targets binder discovery specifically, where template-guided masking of a known binder explores sequence space around a validated scaffold rather than sampling blindly. The metagenomic pretraining corpus makes it a reasonable choice for work on uncultured or non-model organisms whose sequences are underrepresented in curated databases.

#Impact

DSM is one of the clearer demonstrations that masked diffusion resolves the long-standing representation-versus-generation tradeoff in protein language modeling, and it does so cheaply by adapting existing ESM2 weights rather than pretraining from scratch — a recipe other groups can follow. The open release across two scales, with LoRA variants and full training code, makes it directly usable. The binder results in the preprint rest on predicted rather than measured affinities, and the largest checkpoint remains modest at 650M parameters, so absolute performance still trails the largest closed protein models.