Plasma Protein-Token Transformer

A proteomic risk score fitted to proteins A, B, C and D cannot be run on a cohort that measured only A and B. The coefficients for the missing two have no input to multiply, so the model must be rebuilt — which demands outcome labels the new cohort, valuable precisely for its proteomics, often lacks. Since measured protein sets differ across cohorts, study phases and assay configurations, this dependence on an identical feature space is what blocks reuse of population-scale proteomic predictors.

The Plasma Protein-Token Transformer answers it by discarding the fixed feature vector. Each measured protein becomes a token — a learned protein-identity embedding plus a linear projection of that protein's standardized abundance — and a classification token prepended to the sequence pools them into a single 256-dimensional participant representation. A protein that was never assayed is simply absent from the sequence rather than imputed to a cohort mean, so the same encoder accepts a 2,920-protein profile and a 1,460-protein profile and returns a vector of identical shape from both. Pretraining is masked-protein reconstruction: 15% of the proteins visible in a sampled coverage view have their abundance projection swapped for a mask embedding while the identity embedding is kept, and the model is optimized to recover the masked abundance.

The encoder was developed at the University of Pennsylvania by Yonghyun Nam, Dokyoon Kim and colleagues and posted to medRxiv in September 2026. It carries into the plasma proteome the token-based self-supervised recipe that Geneformer and scGPT established for single-cell transcriptomes, where the same variable-length, set-valued input problem arises.

#Key Features

• Omission instead of imputation: Proteins with no measured NPX value are dropped from the input sequence rather than filled in, so a representation never absorbs a synthesized value standing in for an assay that was not run.
• Frozen encoder, swappable heads: Encoder parameters are frozen after pretraining and no disease-specific fine-tuning is performed; each disease gets its own L2-regularized logistic head reading the same representation.
• Coverage-varying pretraining: Every minibatch samples a coverage view — the full 2,920 proteins, a random 800–2,500 subset, or the predefined 1,460-protein panel — so the encoder meets varying availability during training, not only at deployment.
• Label-free transfer: The headline experiment applies the same frozen encoder and disease models to half-coverage profiles with no retraining, recalibration or cohort-specific normalization.

#Technical Details

The encoder is six Transformer layers, eight attention heads, hidden dimension 256 and feed-forward dimension 1,024, over a canonical vocabulary of 2,920 proteins. Pretraining ran 30 epochs with AdamW at an initial learning rate of 10⁻³, weight decay 10⁻², cosine decay, gradient clipping at 1.0 and effective batch size 64. Training data are Olink Explore 3072 plasma profiles from 53,014 UK Biobank Pharma Proteomics Project participants: batches 0–6 ran all eight assay panels for 2,920 proteins after quality control, while batch 7 ran four panels for a nested 1,460-protein subset. 80% of batch 0–6 participants (n=36,380) formed the development cohort, 9,118 were held out, and batch 7's 7,516 participants plus 1,172 repeat-assessment profiles were reserved for evaluation.

Across 144 incident diseases at a three-year horizon, median AUC was 0.679 with all 2,920 proteins, 0.637 when the frozen encoder and unchanged disease models were applied to 1,460-protein representations, and 0.673 when only the disease models were refitted. The L1-regularized comparators scored 0.654, 0.617 and 0.630 respectively. Under partial coverage the token model exceeded coefficient-truncated LASSO by a median paired AUC difference of 0.027 (95% CI 0.018–0.041) and came within 0.003 of a LASSO model refitted using outcome labels.

#Applications

The target users are epidemiological cohorts, biobank sub-studies and clinical proteomics programmes holding plasma measurements on a partial panel but too few outcome events to rebuild a disease model: they can encode what they measured and apply an existing risk model unchanged. Added to age, sex, body mass index, smoking status and ethnicity, the representation raised AUC for all 12 focused cardiovascular-kidney-metabolic and autoimmune endpoints, and for 10 of 12 under partial coverage without refitting.

#Impact

The work reframes proteomic risk prediction as a representation problem rather than a feature-selection one, and shows the reframing costs nothing under full coverage — 0.679 against the sparse baseline's 0.654. Its limits are stated plainly. The comparison uses nested panels from one biobank on one assay platform, and the 1,460-protein subset was among the coverage views seen during pretraining, so this is panel-aware representation learning, not zero-shot transfer to an unseen configuration. Retention is uneven: cardiovascular-kidney-metabolic endpoints lost 0.009–0.029 AUC, while coeliac disease, systemic lupus erythematosus and multiple sclerosis lost 0.070, 0.099 and 0.111. Only discrimination was assessed, not calibration. The preprint has not been peer reviewed, no code or weights accompany it, and the proteomic and outcome data are reachable only through an approved UK Biobank application.