CytoBERT

Two immunology labs can run mass cytometry on the same blood samples and still produce data that no shared model can read. One panel measures 32 antibody markers, the next measures 41, the overlap between them is partial, and the same protein often arrives under a different channel or clone name in each. Deep-learning methods built for cytometry — CellCNN, CytoSet, CytoDx — all consume a fixed-width cell-by-marker matrix, so a new study means a new input space and a model trained from scratch. CytoBERT takes the panel out of the architecture and puts it into the data.

Each cell becomes an unordered sequence of tokens, one per measured marker, and each token embedding is the sum of a learned marker-identity embedding and a learned embedding of that marker's discretized expression level. A panel is then simply which tokens are present, so a study is readable as long as its markers appear in the shared vocabulary, however many of them it measures. Pretraining masks the expression half of a token while leaving the marker identity visible and asks the encoder to recover the missing expression bin from the rest of the cell — masked-language modelling transposed from word order onto within-cell co-expression, in the manner that scBERT and scGPT applied to transcriptomics.

CytoBERT was released in August 2026 by researchers at the University of Rostock, Philipps-Universität Marburg and the Hessian Center for Artificial Intelligence. It adopts the design of ImmuneFM, which the paper describes as lacking a publicly available implementation, and pairs it with a full artifact release: code under MIT, the pretrained checkpoint, the preprocessed corpus and a Docker image.

#Key Features

• Panel-agnostic input: Marker identity and expression are separate embeddings summed per token, so heterogeneous panels are handled natively rather than by intersecting every study down to its shared columns.
• Masked expression modelling: Masking is applied only to non-zero marker positions, on the reasoning that the zeros that dominate cytometry readouts carry little signal to reconstruct.
• Standardized marker vocabulary: A 220-marker vocabulary harmonized across 18 public ImmPort CyTOF studies, released alongside the model, is reusable independently of it.
• Sample-level aggregation: A CLS token yields one embedding per cell; mean pooling over 512 sampled cells yields a patient-level vector that a two-layer MLP head classifies.
• Grouped-subject evaluation: Folds are split by subject rather than by sample file, preventing a model from recognizing a patient instead of a disease.

#Technical Details

The encoder is a 6-layer Transformer with hidden dimension 128, 8 attention heads and a 512-unit feed-forward block — roughly 1.2 million parameters, small by foundation-model standards. Expression values are arcsinh(x/5)-transformed, MinMax-scaled per marker across the corpus and discretized into 10 equal-width bins, with an eleventh embedding reserved for the mask. Pretraining covers approximately 50 million cells from 15 of the 18 curated ImmPort CyTOF studies, masking 25% of eligible non-zero bins each epoch over 50 epochs. Downstream, the encoder and classification head are optimized jointly on seven ImmPort cohorts under 3-fold cross-validation, scored by ROC-AUC. CytoBERT takes the best average rank (1.6) ahead of logistic regression (2.4), CytoSet (2.7) and a randomly initialized CytoBERT (3.3), but a Friedman test across the seven datasets returns p = 0.093: the margin over the supervised baselines is not statistically significant. The clearest gains fall on cohorts excluded from pretraining — 0.723 versus 0.560 for logistic regression on SDY1708 (COVID-19 severity), 0.721 versus 0.593 on SDY997 (lupus nephritis) — while on the 20-subject SDY788 it reaches 0.246 against logistic regression's 0.487.

#Applications

The evaluated task is sample-level clinical classification from a patient's cytometry file: COVID-19 severity and diagnosis, lupus nephritis, HIV status, early-stage liver cancer staging, peanut allergy, and response to kidney-transplant desensitization therapy. It suits the common situation in immunology — a few hundred labelled samples profiled on a panel that matches no other study, where training a classifier from scratch has historically been the only option. The repository also ships tutorials for cell-level classification and reference mapping, though neither is evaluated in the preprint.

#Impact

CytoBERT's contribution is less a performance result than a reproducible one. Its strongest evidence is the ablation against an identically shaped model trained from scratch, and the paper says plainly that the differences from supervised baselines are not significant — candour worth noting in a field where cytometry benchmarks are often reported under sample-level splits that leak subject identity between folds. Downstream use requires end-to-end fine-tuning; no frozen-feature or zero-shot results are reported. The harmonized marker vocabulary and the openly redistributed corpus lower the entry cost for the next cytometry foundation model as much as the checkpoint does. The work is a preprint and has not yet been peer reviewed.