ProteinReasoner

Multi-modal protein language models have converged on a common recipe: tokenize structure, tokenize sequence, fuse both at the input embedding layer, train end to end. ProteinReasoner argues that this discards something protein scientists rely on. AlphaFold 2 does not map sequence to structure in one leap — it routes through co-evolutionary signal from multiple sequence alignments, just as directed evolution routes through rounds of experimental feedback. Evolution acts as an intermediate reasoning step, not an auxiliary feature.

ProteinReasoner makes that step explicit. Developed by BioMap Research with a collaborator at Stanford University and posted to bioRxiv in July 2025, it is a generative model over three modalities — structure, evolutionary profile, and amino acid sequence — with the profile sitting between the other two as a chain-of-thought intermediate. That profile is an L × 21 matrix of per-position amino acid frequencies from an MSA, so the model reasons over a continuous numerical representation, not only discrete tokens.

The design pays off in efficiency. ProteinReasoner-650M beats ESM3-Open 1.4B and DPLM-2-3B on zero-shot structure prediction and inverse folding despite being several times smaller, and its reasoning chain extends to protein optimization: appending profiles built from wet-lab fitness scores lets the model propose improved variants in context, with no weight updates.

#Key Features

• Evolutionary profile as an intermediate modality: Rather than treating MSA statistics as an input feature, the model predicts the profile as a step in its own right, under a KL-divergence loss alongside cross-entropy on structure and sequence tokens.
• Modality separation through the transformer: Sequence, profile, and structure are kept as separate token streams across all layers with modality-specific decoder heads, in contrast to ESM-3, which fuses modalities at the embedding layer.
• Bidirectional pretraining: Batches are split evenly between the folding order (sequence → profile → structure) and its inverse, reinforcing cross-modal associations in both directions.
• Guided and inferred inference modes: Given an MSA profile the model runs in "guided" mode; without one it generates the profile itself first. Guided mode is 9.5–33.8% better in TM-score, quantifying how much MSA quality still matters.
• In-context protein optimization: Directed evolutionary profiles — amino acid frequencies weighted by measured fitness — are appended to the reasoning chain, extending an optimization trajectory without retraining.

#Technical Details

Two sizes were trained: 150M parameters (30 layers, hidden dimension 640) and 650M (33 layers, dimension 1280), both with 20 attention heads and rotary positional encoding. Structure is discretized with the DPLM-2 structure tokenizer (8,192-token vocabulary), sequence uses a 26-token residue vocabulary, and the profile is projected through a linear layer. Pretraining is two-stage: stage 1 is BioMap's sequence-only backbone from its compute-optimal scaling study, trained on 1 trillion amino acid tokens; stage 2 initializes from it and trains on 9.45 million AlphaFold Database structures plus 311k PDB-REDO chains with MSAs, for a budget of 1.89 × 10¹¹ tokens.

In guided mode, ProteinReasoner-650M reaches TM-scores of 0.624 on CASP14, 0.774 on CAMEO, and 0.850 on a PDB date-based holdout, against 0.566/0.747/0.794 for ESM3-Open 1.4B and 0.527/0.711/0.811 for DPLM-2-3B. On inverse folding it achieves the best self-consistency TM-scores of any model tested, 0.786 (CAMEO+CASP) and 0.918 (PDB split), though DPLM-2-3B recovers more native residues. On the ProteinGym DMS substitution benchmark the 150M model reaches 0.489 average Spearman versus 0.466 for ESM3-Open 1.4B; scaling to 650M gives no gain, indicating fitness ranking is less scale-sensitive than the generative tasks.

#Applications

The most concrete application is stability engineering. Fine-tuned with supervised training and direct preference optimization on the Megascale dataset, the in-context learning variant reached Spearman 0.778 and AUROC 0.854 on 63 held-out proteins, ahead of an active-learning baseline that fine-tunes a separate model per target (0.752 / 0.838); the gap widened on double mutants (AUROC 0.841 versus 0.805). Because the approach needs no per-campaign retraining, one checkpoint can serve many parallel directed-evolution projects, consuming assay results as prompt context between rounds. The zero-shot structure prediction and inverse folding modes support the same workflows upstream, generating sequences for a target fold or checking designs before synthesis.

#Impact

ProteinReasoner reframes MSAs for the multi-modal PLM era: not a slow preprocessing dependency to be engineered away, but a supervision target that teaches a model to reason. Its central claim — that structuring modality interactions beats scaling parameters — is backed by ablations showing the profile step adds 23.8–35.8% TM-score in guided mode and 10.9% Spearman on ProteinGym. Several caveats matter. The work is a preprint that has not been peer reviewed, released under CC BY-NC 4.0, and no code or model weights have been published, so the results are not independently reproducible. Internally inferred profiles remain measurably worse than MSA-derived ones, and the in-context optimization framework was validated on a single simulated round over existing deep mutational scanning data, with no prospective wet-lab campaign.