DPLM (Diffusion Protein Language Model)

A masked protein language model is trained to recover roughly 15% of a sequence at a time. That is an excellent way to learn what an amino acid means in context — every prediction sees the whole protein on both sides — but it never asks the model to produce a protein from nothing, so generation has to be improvised afterwards with Gibbs sampling or Monte Carlo tricks. Autoregressive models generate cleanly but read strictly left to right, giving up the bidirectional receptive field that made masked models good at understanding.

DPLM treats both as corner cases of a single objective. Under absorbing discrete diffusion, the forward process replaces amino acids with a mask token at a corruption level drawn anywhere between 0% and 100%, and the model denoises at every level; fixing that level at 15% recovers masked language modeling, and a left-to-right schedule recovers autoregression. Sampling is the reverse process: start from a fully masked sequence of the requested length and iteratively unmask, re-masking low-confidence positions, for a few hundred steps. One set of pretrained weights therefore supports unconditional generation, conditioning on a partial sequence, and embedding extraction.

DPLM was developed by ByteDance Research with the Department of Computer Science at Nanjing University, posted as a preprint in February 2024 and published at ICML 2024. It is the sequence-only origin of the family extended to structure by DPLM-2 and refined by DPLM-2.1, all three distinct checkpoints in one repository. It is unrelated to the dynamics-aware DPLM, which shares only the acronym.

#Key Features

• Denoising at every noise level: Training spans corruption ratios from clean to fully masked rather than a single fixed ratio, which is a harder reconstruction problem and, the authors argue, why the resulting representations transfer better than ESM-2's.
• Two-stage pretraining: Weights are first trained under a masked language modeling objective, then continued under the diffusion objective. Training the diffusion objective from scratch optimizes poorly; the staged recipe is what lifts generated pLDDT close to 90.
• Sequence-conditioned scaffolding: Because partial-sequence conditioning is already learned during pretraining, a functional motif can be pinned in place and the scaffold sampled around it with no fine-tuning.
• Cross-modal conditioning by adapter: Freezing the backbone and training a small cross-attention adapter over a GVP-Transformer structure encoder turns the same checkpoint into an inverse-folding model.
• Plug-and-play classifier guidance: A first-order Taylor approximation makes classifier guidance well defined over discrete tokens, so an external secondary-structure classifier can steer sampling toward a target fold without touching the weights.

#Technical Details

Pretraining uses UniRef50 — about 45 million sequences, 14 billion amino acid tokens — with long proteins truncated to random 1,024-token windows. Architecture and scales follow ESM-2 exactly, giving 150M, 650M and 3B checkpoints trained for 100K updates at 320K tokens per batch (150M) and 1M tokens per batch (650M and 3B). Unconditionally sampled sequences average ESMFold pLDDT above 80 from length 100 to 1,000, and scale helps most at the long end: at length 1,000 the 3B model holds 83.86 where the 150M model falls to 72.10. As a representation learner the 650M model exceeds ESM-2 650M on every task in the paper's suite — thermostability Spearman ρ 0.695 against 0.691, metal ion binding 75.15 against 71.88, EC Fmax 0.875 against 0.866 — and the 3B model reaches EC 0.883, approaching SaProt without being given structures. On 17 motif-scaffolding problems it solves more than EvoDiff and beats RFDiffusion on six, notably the calmodulin EF hand and the carbonic anhydrase II binding site, while losing on rigid helical motifs. With a GVP-Transformer adapter, inverse-folding amino acid recovery on CATH 4.3 is 53.27% at 150M, 56.61% at 650M and 59.44% at 3B, at self-consistency TM-score 0.85–0.86.

#Applications

The checkpoints suit de novo sequence design where no backbone is specified, scaffolding around a known functional motif, and property prediction from fine-tuned embeddings — three uses that would otherwise need separate generative and encoder models. Secondary-structure-guided sampling gives designers coarse control over the fold without training a conditional model, and the adapter recipe extends conditioning to backbone structure. Apache-2.0 code and openly downloadable weights at three sizes make the 650M checkpoint a practical starting point for fine-tuning.

#Impact

DPLM's contribution is the framing: masked and autoregressive protein language modeling are endpoints of a discrete diffusion spectrum, and training across that spectrum improves generation and representation together rather than trading one for the other. ByteDance built directly on it — DPLM-2 warm-starts from these weights because structure data is too scarce to pretrain a multimodal model from scratch. The limitations are real: the model handles sequence only, motif-scaffolding success rates are modest, the 1,024-token training window bounds context, and every result is computational, scored with ESMFold and OmegaFold rather than by experimental characterization.