CytoFormer

Within a single hematoxylin and eosin (H&E) nucleus, a macrophage, a fibroblast and a poorly differentiated tumour cell can look almost the same. That ambiguity is why cell typing on routine histology has stalled: the expert-annotated resources — PanNuke, CoNSeP, MoNuSAC — hold on the order of 100,000 nuclei, use a few coarse categories, and disagree on what those categories mean. Scaling annotation cannot fix a label a pathologist cannot reliably assign by eye.

CytoFormer changes the supervision rather than the architecture. Imaging-based spatial transcriptomics measures hundreds to thousands of transcripts per cell in situ and segments individual cells; the same section can then be stained with H&E. Molecular identity and morphology are therefore observed for the same physical cell, and the molecular channel can label the morphological one. Developed at the University of Pennsylvania by Jialu Yao, Songhao Li, Alina Yu and Zhi Huang and posted in August 2026, it was trained this way on 15,422,352 cells from 81 paired Xenium/H&E sections spanning 16 organs, with no image annotated by hand.

It also works at a different granularity from the pathology foundation models it builds on and is compared against. UNI2-h, Virchow2, MUSK, CONCH, PathGen-CLIP and PLIP encode tiles or whole slides; CytoFormer encodes one nucleus and its immediate neighbourhood, returning a map of typed cells across an entire section.

#Key Features

• Molecular supervision in place of manual annotation: Reference labels come from Leiden clustering of Xenium expression profiles, marker-gene annotation, organ-wise human review and a per-cell confidence gate, then transfer onto the registered H&E morphology.
• A field of view chosen by ablation: The model reads a 56 micrometre crop centred on the nucleus, resized to 224 x 224 pixels. Cropping identical cells at 28, 56, 112 and 224 micrometres gave macro-F1 of 79.1, 83.3, 82.5 and 77.1 — enough context to separate stroma from endothelium without losing diagnostic resolution.
• Shared encoder, per-organ heads: Sixteen linear classifiers sit on one jointly trained encoder, 103 outputs in total. The organ identifier only routes to a head and never reaches the encoder, so morphology recurring everywhere is learned from all tissues at once.
• A reusable cell embedding: extract_features returns the organ-independent 1,536-dimensional representation directly, which is what the linear-probing and active-learning experiments use.
• Spatially blocked evaluation: Training and test cells split by location rather than at random, with a 112 micrometre guard band discarding 366,227 boundary cells so no test crop shares a field of view with a training crop.

#Technical Details

The encoder is a ViT-giant with patch size 14 and a 1,536-dimensional embedding, initialised from UNI2-h's 681-million-parameter weights and fine-tuned end-to-end on 12,111,769 training cells with AdamW at batch size 256 in bfloat16 across four NVIDIA B200 GPUs; the retained checkpoint is from epoch 5. On the 2,944,356-cell spatially held-out split it reaches 0.846 accuracy and 0.779 macro-F1 across all 16 organs, with per-class F1 spanning hepatocyte at 98.1 down to chondrocyte at 37.6. Frozen-feature transfer tests the representation: with a linear head, CytoFormer gave the best result in 24 of 25 dataset-by-organ settings on PanNuke, CoNSeP, MoNuSAC and PUMA — four expert-annotated datasets independent of its training data — averaging 79.2 macro-F1 against 71.4 for the second-best method in each setting. Nine of PanNuke's 19 organs were absent from pretraining, and it led on every one.

#Applications

Typing every nucleus turns judgements pathologists report semi-quantitatively — tumour cellularity, lymphocyte infiltration density, macrophage and plasma-cell abundance, the state of surrounding stroma and vasculature — into counts reproducible across slides and cohorts. The released pipeline takes a whole-slide image plus a table of nucleus centroids and returns a per-cell prediction table; a browser viewer shows the output on two held-out sections. The embedding is useful on its own too: on a colorectal VisiumHD section, a logistic head on CytoFormer features detected benign epithelium amid look-alike adenocarcinoma at F1 0.82 against 0.69 for the strongest baseline, reaching most of its accuracy within about 200 annotations.

#Impact

The result worth carrying forward is methodological: paired spatial transcriptomics can substitute for pathologist annotation as the supervision signal for cell-level morphology, and a model trained that way generalises to organs and cell types outside its own label space. The limits are equally concrete. The taxonomy is deliberately coarse — T and B cells merge into a single lymphocyte class, macrophage states are not separated — and the labels inherit Xenium segmentation and transcript spill-over error rather than pathologist judgement. Mononuclear classes are the weak point: on a held-out breast section, 44.6% of macrophages and 63.5% of plasma cells were called lymphocytes. The classification head requires an organ identifier, though the encoder does not. This is a preprint awaiting peer review, the curated 15.4-million-cell dataset has not been released, and both the code and the checkpoint carry non-commercial licenses.