PALM

Amyloid fibrils are filamentous, cross-β protein aggregates implicated in Alzheimer's disease and type-2 diabetes, and they are also a practical obstacle to drug development: an aggregation-prone region (APR) buried in a candidate peptide can wreck its solubility, stability, and pharmacokinetics. Characterizing which sequences aggregate is slow and expensive laboratory work, so the labeled datasets available for supervised learning are tiny — WaltzDB-2.0, the field's standard training set, holds just 1,416 hexapeptides, of which 515 form fibrils.

PALM (Predicting Aggregation with Language Model embeddings), from the Molecular AI group at Novo Nordisk, attacks that data scarcity with transfer learning. Instead of learning aggregation from sequence alone, PALM freezes a pretrained protein language model — ESM-2 — and trains a small task head, the Aggregation Predictor Module (APM), on its residue embeddings. It was posted as a preprint in September 2025 and published in the Journal of Chemical Information and Modeling in 2026.

The architecture yields a property most aggregation classifiers lack. PALM is trained only on sequence-level amyloid/non-amyloid labels, yet because the sequence score is a weighted mean over per-residue scores, APR localization emerges without any residue-level supervision. It sits alongside other embedding-based aggregation probes such as the Bi-LSTM/Bi-GRU amyloidogenicity predictor, and is distinguished by embedding whole sequences rather than sliding hexapeptide windows.

#Key Features

• Residue-level APR maps from sequence-level labels: A softmax-weighted mean over per-residue scores makes the sequence prediction differentiable with respect to each position, so aggregation hotspots are localized without residue annotations in training.
• Padding augmentation bridges hexapeptides and proteins: WaltzDB hexapeptides are flanked with 1–10 sampled non-hydrophobic residues per terminus and oversampled tenfold to 14,148 sequences, aligning their embedding distribution with the longer natural sequences seen at inference.
• Smaller language model wins: Across ESM-2 at 8M, 35M, 150M, and 650M parameters, the 8M backbone performed best, inverting the scaling trend seen for structure prediction.
• Two trained variants: The WaltzDB-2.0 model and a variant retrained on the 100,730-peptide CANYA NNK1-3 library cover the small-data and large-data regimes; both ship as five-fold cross-validation ensembles.
• Released weights under a permissive license: The MIT-licensed repository includes checkpoints for all variants plus a CLI for FASTA, CSV, and direct-sequence inference with residue-score plots.

#Technical Details

The APM applies two independent 1D convolutions (kernel size 5, stride 1) to the ESM-2 embedding tensor to produce value and attention tensors, softmaxes the attention over sequence length, multiplies elementwise, and passes the result through a two-layer MLP (hidden width 32) with dropout 0.25, ReLU, and batch normalization. A sigmoid yields per-residue scores; a softmax-weighted mean over them gives the sequence score. Training minimized binary cross-entropy with SGD at learning rate 0.05, batch size 1000, and early stopping.

On Serrano157, PALM reached ROC AUC 0.918 ± 0.003 and AUPRC 0.770 ± 0.013, ahead of TANGO (0.894), AggreProt (0.888), ANuPP (0.851), CANYA (0.822), AggreScan (0.817), and Waltz (0.794). On residue-level AmyPro22 it scored 0.678 ± 0.018, competitive with the same baselines. The paper reports a clear failure mode: on 753 single-substitution Aβ42 mutants, PALM scored 0.510 ± 0.030 — chance — because wild-type residue scores were already saturated near 1, while TANGO (0.741) and CANYA (0.738) succeeded. Retraining on CANYA NNK1-3 raised NNK4 performance to 0.824 ± 0.002 and Aβ42 to 0.622 ± 0.018, with a one-hot-encoded variant reaching 0.705.

#Applications

PALM is aimed at peptide and biologic drug discovery, where developability triage happens before molecules reach the bench. Given a candidate sequence, it returns both an overall aggregation probability and a residue-score profile that points to which segments drive it, letting protein engineers target specific positions for redesign rather than discarding a whole candidate. The residue profiles also support mechanistic analysis of natural amyloidogenic proteins, reproducing annotated APRs in Aβ42 and influenza PB1-F2.

#Impact

PALM is a useful data point for how far transfer learning carries a task with scarce labels: a 6-layer language model plus a lightweight head matches or beats biophysics-based tools trained on decades of domain knowledge. Its honest negative result is arguably as valuable — pLM embeddings plus 1,416 training peptides cannot resolve single-mutation effects, and closing that gap took two orders of magnitude more experimental data. The scores are best used for ranking and hotspot triage rather than absolute classification, since residue-score distributions shift between datasets. Weights, training data, and evaluation sets are all public under MIT, making the result directly reproducible.