LSD

Protein sequence representation learning is dominated by masked language modelling: mask a fraction of residues, predict them, and take the encoder's hidden states as embeddings. Latent Space Diffusion (LSD) asks whether that single objective can usefully be split in two. The framework separates manifold learning — finding a continuous latent space that faithfully encodes protein sequences — from distributional modelling — learning how real sequences are distributed within that space. An autoencoder handles the first, and a denoising diffusion model trained on the autoencoder's latents handles the second.

The payoff of that split is that the diffusion model does not yield one embedding but a one-parameter family of them, indexed by the diffusion time t. Each point on that family mixes the latent code with the model's learned denoising direction in a different proportion, so a practitioner can sweep t and pick the representation that suits a downstream task rather than accepting whatever the pretraining objective happened to produce.

LSD was released as a preprint in March 2025 by a team at InstaDeep. It is a controlled methodological study rather than a production model: the variants are small, the comparison is against a masked-language-model baseline trained in the same pipeline, and the finding is reported in both directions.

#Key Features

• Two-stage decomposition: A transformer encoder maps a tokenized sequence to per-residue latent vectors and a decoder reconstructs token logits; a separately trained conditioned transformer then models the distribution over those latents.
• Diffusion-time embedding family: Combining the diffusion model's prediction with the latent code, weighted by a function of the diffusion time, produces a continuum of representations from a single trained model.
• Token-norm variant (LSD-TN): A homogeneous autoencoder that applies a separate normalization loss per amino acid type, forcing residues of the same type to be identically distributed in latent space.
• Noise-masking variant (LSD-NM): An inhomogeneous autoencoder that replaces discrete masking with position-varying Gaussian noise, weighting the reconstruction loss by how corrupted each position was.
• Frozen-backbone evaluation: All downstream results use frozen representations averaged across the sequence and fed to a single-hidden-layer MLP, isolating representation quality from fine-tuning capacity.

#Technical Details

The diffusion component uses a variance-preserving cosine noise schedule with a v-target objective. Two sizes were trained: a small configuration with a 4.7M-parameter autoencoder and a 7.3M-parameter diffusion model, and a medium configuration with an 18.9M-parameter autoencoder and a 29.0M-parameter diffusion model. Pretraining used UniRef50 sequences truncated at 254 residues plus start and end tokens, dropping the 0.5% of sequences containing non-canonical amino acids, with AdamW at a learning rate of 2e-5, weight decay 1e-3, and batch size 512 on a single A100 — 200k steps for the small models and 100k for the medium.

Evaluation covers five property-prediction tasks adopted from SaProt: thermostability (Spearman's ρ), HumanPPI, metal ion binding, and the ten-class and binary DeepLoc localization tasks (accuracy). The strongest diffusion representation, LSD-NM-M, reaches ρ = 0.581, 61.1% on HumanPPI, 64.7% on metal ion binding, 54.2% on DeepLoc-Subcellular, and 76.8% on DeepLoc-Binary. Both LSD variants beat the diffusion model trained over a masked-language-model latent space, which reaches ρ = 0.474 at small scale and 0.543 at medium. None of the diffusion representations match the masked-language-model embeddings themselves: ESM-2 at 650M parameters reaches ρ = 0.690, 81.3% on HumanPPI, and 91.0% on DeepLoc-Binary, and DPLM at the same scale is comparable. Those baselines are more than twenty times larger than the largest LSD configuration.

#Applications

The immediate use is as a research instrument for teams designing protein representation objectives. The two autoencoder variants isolate a specific question — how the latent space should be regularized so a generative model trained on top of it produces discriminative features — and the frozen-probe protocol across thermostability, protein-protein interaction, metal ion binding, and subcellular localization gives a reusable way to answer it. The latent spaces themselves are also a substrate for generative work, since a diffusion model operating on continuous latents sidesteps the discrete sampling machinery that sequence-space diffusion models require.

#Impact

LSD contributes a clean, honestly reported result on a question the field tends to answer by assertion: latent-space regularization choices measurably change how discriminative a diffusion model's representations are, and both proposed regularizers beat the masked-language-model latent space under identical training. At the same time, the paper reports that no LSD representation reaches the quality of masked-language-model embeddings, which keeps the practical recommendation with established protein language models for now. The comparison is against much larger baselines, so the gap conflates approach with scale, and the study evaluates only discriminative transfer — generation quality from the latent diffusion model is not assessed. No code, weights, or license accompany the preprint, which has not been peer reviewed.