DPLM-2.1

DPLM-2.1 is a multimodal diffusion protein language model that generates and represents amino acid sequences and 3D backbone structures inside a single discrete-diffusion transformer. It was developed by ByteDance Seed with collaborators at Nanjing University and Rutgers University, and presented as a Spotlight paper at ICML 2025 under the title "Elucidating the Design Space of Multimodal Protein Language Models."

Multimodal protein language models such as ESM3 and DPLM-2 handle geometry by quantizing 3D coordinates into discrete structure tokens, so one transformer can read and write both modalities. DPLM-2.1 begins with a diagnosis of why that pipeline underperforms specialist folding networks. Quantization is itself lossy — round-tripping a structure through the tokenizer degrades reconstruction from 1.31 Å to 1.98 Å RMSD — but the larger bottleneck is that the language model predicts structure tokens badly. Index-level cross-entropy treats semantically adjacent codes as unrelated categories, and a 650M DPLM-2 recovers the correct structure-token index only 11.9% of the time.

Rather than proposing a single new architecture, the work maps a design space spanning generative modeling, structure-aware architecture, representation learning, and training data, and reports which choices compose and which are redundant. The headline result is that a 650M-parameter model assembled from the recommended combination reduces folding RMSD on the PDB date-split test set from 5.52 Å to 2.36 Å, overtaking 3B-parameter baselines.

#Key Features

• Bit-level structure supervision: Instead of choosing one index from a large structure-token vocabulary, the model predicts each bit of the quantized structure code. This raises structure-token index accuracy from 0.1188 to 0.2641 and bit accuracy from 0.7932 to 0.8648, and improves folding RMSD on the PDB date split from 5.31 Å to 3.22 Å on its own.
• Geometry-aware architecture: Pair bias, structure transition layers, and triangular update and attention modules inject the geometric inductive biases that plain sequence transformers lack, and combine cleanly with bit-level supervision.
• Representation alignment: Aligning the model's internal representations with those of a pretrained folding network provides a smooth, high-dimensional learning signal alongside the discrete token objective, improving both folding accuracy and generation diversity.
• Single checkpoint, three tasks: The same weights perform unconditional sequence-structure co-generation, forward folding, and inverse folding without task-specific heads.
• Multimer-aware training data: A curated PDB-Multimer set of multi-chain structures is used to study monomer/multimer transfer, which improves modeling for both single- and multi-chain proteins.

#Technical Details

The released DPLM-2.1 checkpoint, dplm2-bit-650m, is a 650M-parameter ESM-style transformer with 33 layers, hidden size 1280, and 20 attention heads, trained on the same PDB- and SwissProt-derived structure corpus as DPLM-2 under an absorbing discrete-diffusion objective. With bit-based modeling alone it reaches 3.22 Å RMSD and 0.904 TM-score on the PDB date split and 6.40 Å / 0.838 on CAMEO 2022, against DPLM-2 650M's 5.31 Å / 0.831 and 7.70 Å / 0.794. Adding geometric modules brings the PDB date split to 2.55 Å / 0.925 while raising unconditional generation diversity from 0.700 to 0.900, and the best folding-tuned configuration reaches 2.37 Å. For reference, ESMFold at 3B parameters scores 2.84 Å on the same split, so DPLM-2.1 matches specialist folding accuracy there while still trailing on CAMEO 2022. Inverse-folding amino acid recovery on CAMEO rises to 0.5586, above both DPLM-2 650M (0.4962) and DPLM-2 3B (0.5236). The bit-based checkpoint and the preprocessed training dataset are on HuggingFace, and the shared DPLM monorepo is Apache-2.0; the repository currently implements the bit-based component, with the geometric and representation-alignment modules still to be released.

#Applications

DPLM-2.1 targets researchers who need sequence and structure produced together rather than in sequence. Because one model covers co-generation, folding, and inverse folding, it supports de novo backbone design followed immediately by sequence assignment, structure-conditioned sequence redesign, and rapid structural screening of designed sequences — all at 650M parameters, which fits comfortably on a single GPU. The design-space analysis is also directly reusable: the tokenizer diagnostics, bit-level supervision recipe, and alignment strategy transfer to any token-based multimodal protein model.

#Impact

The paper's contribution is as much methodological as it is a checkpoint: it establishes that discrete structure tokens are not inherently the ceiling on multimodal protein language models, and that most of the gap to specialist folding networks comes from how the tokens are supervised rather than from tokenization alone. Demonstrating that a 650M model can outperform 3B baselines reframes scale as a secondary lever for this class of model. The authors are explicit about what remains unsolved: discrete representations still discard continuous geometric variation and operate at residue or backbone rather than atomic resolution, the models carry no explicit physical or energy-based priors, and the analysis stops at 3B parameters, leaving the scaling behavior of these design choices untested.