ImmuneFM

Two immunology labs can profile the same tube of blood and produce data that no single model can read: one panel stains CD3, CD4 and CD8; the next adds CD127 and CD25 and drops CD11b. Because the columns of a cytometry matrix differ from study to study, sample-level classifiers are rebuilt from scratch for each cohort — and a typical ImmPort study carries only 70 to 260 labeled samples.

ImmuneFM fixes a shared vocabulary instead. The authors hand-curated 203 panel markers that recur across public cytometry studies and gave each one its own learnable embedding, so any study contributes whatever subset of that vocabulary it measured. Each cell becomes a sequence of 203 marker slots holding discretized expression values, and a Transformer encoder is pretrained to regenerate masked entries from the unmasked ones. Marker order carries no meaning, so there is no NLP-style positional encoding — the "Marker2Vec" table is keyed on marker identity alone.

The model comes from the Bakar Computational Health Sciences Institute at UCSF, the group that also maintains ImmPort itself. Its binning-and-masking recipe is adapted from single-cell transcriptomic language models such as scBERT, scaled down to the few hundred protein markers a cytometry panel measures. It sits alongside broader immune-system models like EVA, but works from protein-level single-cell measurements rather than transcriptomics and histology.

#Key Features

• Shared 203-marker vocabulary: A hand-curated set of commonly used panel markers lets 53 heterogeneous ImmPort studies be pooled into one training matrix — the step that makes corpus-scale pretraining possible at all.
• Masked-marker self-supervision: A quarter of the non-zero entries in each cell's marker vector are masked and reconstructed with cross-entropy loss, requiring no labels from any contributing study.
• Sample-level pooling: Every cell in a sample is encoded by the pretrained backbone and mean-pooled into one sample vector, which a small MLP head classifies — the granularity clinical cytometry questions are actually asked at.
• Cell- and marker-level interpretation: Gradient-based class activation mapping scores how much each cell and marker contributed to a prediction, recovering known immunology and surfacing candidate biomarkers.
• Robust to shallow sampling: Accuracy holds at 128 cells per sample instead of 512, where the FlowSOM clustering baseline degrades sharply.

#Technical Details

The backbone is a six-layer Transformer encoder with model dimension 128, eight attention heads and feed-forward width 512 — roughly 1.2 million parameters, small because a 203-marker panel is orders of magnitude narrower than a transcriptome. Raw intensities are arcsinh-transformed, min-max scaled and binned into 10 discrete levels before embedding. Pretraining ran 100 epochs at learning rate 1e-4 with batches of 1,024 cells on UCSF's Wynton HPC cluster, over more than 100 million single cells drawn by sampling 10% of cells per subject across the 53 studies — vaccine response, autoimmune disease, oncology, transplantation, infection, allergy, preterm birth and general immune response. Fine-tuning adds an MLP head and runs 50 epochs at batch size four.

Evaluation covers eight ImmPort studies of 72 to 259 samples each: liver cancer staging, peanut allergy, lupus nephritis, COVID-19 severity and diagnosis, HIV diagnosis, kidney transplant desensitization, and a mouse cancer-treatment cohort testing cross-species transfer. Four — SDY1733, SDY997, SDY1708 and SDY1108 — are held out of pretraining entirely. ImmuneFM records the best AUROC or balanced accuracy on all eight, ahead of a cytometry CNN, an identically configured Transformer trained from scratch, and a FlowSOM+GBDT feature-engineering pipeline; the last ties it on the transplant task, so the margin over classical gating-style analysis is real but not wide.

#Applications

The target use is sample-level diagnosis and stratification from a cytometry run — deciding whether a blood sample is COVID-19 positive, staging hepatocellular carcinoma from PBMCs, or predicting response to transplant desensitization therapy — where a group has a cohort of only a hundred or so subjects. The interpretation layer makes it usable for discovery as well: in the COVID-19 case study it weighted CD39 and CD64 heavily and traced a neutrophil expansion from 27% to 46% of cells in positive samples, and in the HIV cohort it recovered reduced CD7 alongside elevated KI67 and NK-cell fractions.

#Impact

ImmuneFM treats cytometry — the most widely run assay in immunology, and roughly 70% of ImmPort's experimental results — as a corpus to pretrain on rather than a per-study analysis problem. The curated 203-marker matrix is arguably as significant as the checkpoint, being reusable for training other models. The work remains a preprint awaiting peer review, the released code repository carries no license or documentation, and half the downstream studies overlap the pretraining corpus, so the evidence for generalization rests on the four held-out cohorts. Publishing the pretrained weights alongside the code makes the comparison the paper leans on — pretrained backbone versus the same architecture trained from scratch — directly reproducible by others.