ESMDiff

A protein's function usually lives in the states it moves between — a kinase that opens and closes, a transporter that flips, a disordered region that samples a cloud of shapes. Deep generative models have become the fast alternative to molecular dynamics for sampling those states, but nearly all of them diffuse in 3D Euclidean space. That choice carries two costs: the denoiser must respect rotational and translational symmetry, which constrains the architecture, and an L2 denoising objective rewards local geometry, so the sampler tends to jitter around the basin it started in rather than crossing to a genuinely different mode.

Structure Language Modeling (SLM) sidesteps both by moving the generative problem out of coordinate space. A discrete variational autoencoder first quantizes a backbone into a per-residue string of structure tokens; a conditional language model then learns the distribution over those token strings given the amino acid sequence; a learned decoder maps sampled strings back to 3D. The coarse latent alphabet discards high-frequency geometric detail and keeps the low-frequency conformational variation the ensemble question is about, and a language model inherits no equivariance constraints at all.

ESMDiff is the framework's flagship instantiation, from a Mila-led Canadian collaboration spanning Université de Montréal, McGill, the University of Ottawa, the National Research Council Canada and HEC Montréal. It fine-tunes ESM3 — a bidirectional, BERT-style protein foundation model — with a masked discrete diffusion objective, converting a masked language model into a conditional generator over structure tokens. It is a sibling of the same group's earlier Str2Str, which attacks the same problem by perturbing and annealing 3D coordinates rather than by language modeling. The work appeared as a preprint in October 2024 and was published at ICLR 2025.

#Key Features

• Sequence-conditioned ensembles from one checkpoint: One fine-tuned model samples conformations for equilibrium dynamics, apo/holo pairs and disordered proteins with no per-target refitting.
• Structure as a discrete language: The frozen ESM3 dVAE tokenizer encodes local backbone geometry into residue-level tokens, turning conformational sampling into masked token prediction.
• Masked discrete diffusion fine-tuning: Training optimizes a continuous-time conditional ELBO over mask schedules, with the mask token zeroed out of the output and unmasked tokens copied through unchanged.
• Two sampling modes: Iterative decoding and a DDPM-style reverse process trade diversity against fidelity; 25 denoising steps is the reported default.
• Order-of-magnitude speedup: 20–100x faster than 3D-geometric diffusion baselines, with the gap widening on longer chains.

#Technical Details

ESMDiff is a 1.4B-parameter model built on the open ESM3 backbone. The structure tokenizer is ESM3's pretrained dVAE, frozen throughout; only the transformer and a freshly initialized output head over an augmented structure vocabulary are trained. Training data is 112.4k monomeric PDB chains deposited on or before 1 May 2020, filtered to 5.0 Å maximum resolution and 10–1000 residues — the same cutoff used by AlphaFlow, Str2Str and ESM3, so that comparisons are like for like. Two non-diffusion SLM instantiations, S-T5 (384M) and S-GPT (961M), serve as architecture ablations within the same framework.

On the BPTI benchmark, scored against D. E. Shaw's millisecond MD trajectory, the DDPM sampler reaches the lowest Jensen-Shannon divergence of any method compared on pairwise distance (0.372) and on time-lagged independent components (0.420), with 94% clash-free samples. On the hardest of the five kinetic clusters it recovers a 2.198 Å best-match RMSD. Across 77 fold-switching pairs it attains the best global residue-flexibility correlation (r = 0.402), and on 114 disordered-protein ensembles from PED its radius-of-gyration error is the strongest in the SLM family. MSA-based samplers such as AlphaFlow still lead on several apo/holo metrics, which the authors attribute to the prior that multiple sequence alignments carry.

#Applications

Conformational ensembles answer questions a single predicted structure cannot: whether a cryptic pocket opens often enough to be druggable, how a binding site rearranges between apo and holo states, how compact a disordered region is in solution, and which residues carry tunable flexibility. Because ESMDiff needs only a sequence and is orders of magnitude cheaper than a converged MD run, it is usable as a screening stage — sampling many targets to decide which few deserve simulation — and as a source of receptor conformations for ensemble docking.

#Impact

ESMDiff's broader argument is methodological: a pretrained masked protein language model can be converted into a conditional generative model over structures with a diffusion objective and modest fine-tuning, without retraining a tokenizer or building an equivariant network. The released code is written as a general fine-tuning pipeline rather than a single-purpose script. The limits are visible in the paper's own tables: alignment-based methods remain competitive or better on conformational-change pairs, the model is trained on crystallographic snapshots rather than trajectories so its samples are not Boltzmann-weighted, and evaluation is retrospective against existing MD and NMR ensembles, with no new experimental validation.