ISM (Implicit Structure Model)

Protein language models trained purely on sequence, such as ESM-2, have become the default feature extractor for downstream protein tasks because they need nothing but an amino acid string. That convenience comes at a cost: on tasks that hinge on tertiary geometry — contact prediction, fold-level structure prediction, thermodynamic stability — they trail models that take a resolved or predicted structure as an additional input. Models such as SaProt and ProstT5 close that gap by feeding structure tokens in at inference time, which reintroduces the structural data pipeline that sequence models were meant to avoid.

ISM (Implicit Structure Model), from a team at the University of Texas at Austin, takes the opposite route: it pushes structural information into the weights during training so that inference stays sequence-only. The authors introduce structure-tuning, a self-supervised objective in which a pretrained ESM2 checkpoint learns to predict, for every residue, a discrete token describing that residue's local three-dimensional environment. The tokens themselves come from an atomic autoencoder trained on experimental structures, so what is distilled is fine-grained chemistry rather than a coarse fold label of the kind used by earlier structure-aware variants like S-PLM.

The practical consequence is that ISM ships in ESM2's own weight format. Integrating it into an existing application requires changing a single line of code, which makes it usable as a direct substitute anywhere ESM2 embeddings are already being consumed.

#Key Features

• Sequence-only at inference: No MSA, no predicted structure, and no structural preprocessing are required — the model reads an amino acid sequence and emits structurally informed per-residue embeddings.
• Microenvironment structure tokens: The supervisory signal is derived from every atom within 10 Å of a residue's alpha carbon, not just backbone geometry, so the tokens encode side-chain packing and local chemical interactions.
• Residue-level distillation: Structure-tuning supervises each residue independently, a finer-grained target than fold-classification or CLIP-style sequence-structure alignment objectives.
• Drop-in ESM2 compatibility: Weights are released in both the fair-esm and HuggingFace formats, preserving ESM2's 33-layer, 650M-parameter architecture and tokenizer.
• Two release variants: ISM-650M-UC30PDB is the general-purpose checkpoint; ISM-650M-UC30 excludes PDB structures from training for benchmarking work where PDB leakage is a concern.

#Technical Details

The Atomic Autoencoder pairs a four-layer graph transformer encoder with a two-layer transformer decoder. It takes a masked microenvironment — all atoms within 10 Å of a residue's alpha carbon, with that residue's own atoms removed — encodes element types and pairwise distances, and reconstructs all atomic coordinates under a Kabsch-aligned MSE loss plus a binned distogram loss. Trained on 35,000 PDB proteins, its pooled residue embeddings are quantized by k-means into K = 64 structure tokens, with a second vocabulary of 512 tokens taken from the MutRank evolutionary model. Structure tokens are then generated for 5.8 million Uniclust30 proteins using AlphaFold-predicted structures.

Structure-tuning fine-tunes the 650M-parameter ESM2 for 20 epochs under the sum of the structure-token and masked-language-modeling losses, with a batch of 1,536 proteins cropped to 512 residues, AdamW at a learning rate of 1e-4, and 26 wall-clock hours on 32 GH200 GPUs. Evaluated as a frozen feature extractor, ISM raises CAMEO structure prediction from 0.64 to 0.67 GDT-TS over ESM2 and long-range contact precision from 0.35 to 0.49, matching SaProt's 0.48 despite taking no structure as input. On S669 single-mutation stability, ISM reaches 0.76 AUC against ESM2's 0.72, equalling specialized atomic-environment models at 0.75.

#Applications

ISM is aimed at anyone already running ESM2 embeddings through a downstream head: protein engineers ranking stabilizing point mutations before a round of assays, groups predicting binding residues or secondary structure, and pipelines that fold single sequences without alignments. Because the weights are format-compatible, existing trained heads and evaluation code can be re-run against ISM without modification, making the swap a cheap experiment rather than a migration.

#Impact

ISM demonstrates that the gap between sequence-only and structure-input protein models is partly an artifact of the training objective rather than an inherent limit of sequence models, and it does so without giving up the operational simplicity that made protein language models popular. Distributing the result as ESM2-shaped weights under an MIT-licensed codebase lowers the cost of adoption to near zero for the large body of tooling built on ESM2. The work remains a preprint awaiting peer review, and the released model card is a brief quickstart rather than full documentation of intended use, factors, and evaluation.