CelloType

A spatial omics pipeline conventionally runs in two stages: a segmenter draws cell boundaries, then a separate classifier reads the pixels inside each mask and assigns a cell type. Nothing passes between them. The classifier never sees the surrounding texture that told the segmenter where the boundary was, and a bad boundary — a macrophage's irregular processes clipped off, two apposed lymphocytes merged into one object — arrives as a fait accompli. CelloType collapses the two stages into a single network. One shared transformer encoder produces the latent features from which detection boxes, instance masks and class labels are all predicted, and a composite loss over all three trains them together, so class-specific morphology sharpens the boundaries and the boundaries sharpen the class call.

The model chains three components. A Swin Transformer extracts multiscale features from the image; a DINO detection module turns those features into instance-specific latent features, bounding boxes and class logits using mixed query selection, deformable attention and contrastive denoising; and a MaskDINO module fuses the backbone features with DINO's query embeddings into refined instance masks. Because detection anchors rather than pixel-wise semantic maps carry the objects, one network handles very different scales at once — small hematopoietic cells, large adipocytes, and non-cellular microanatomy such as trabecular bone fragments.

CelloType was developed by Kai Tan's group at Children's Hospital of Philadelphia, with collaborators at the University of Pennsylvania and the University of Iowa, posted as a preprint in September 2024 and published in Nature Methods that November. It sits beside instance segmenters such as Cellpose2 and CellSAM, and is distinguished from them by emitting a cell type along with each mask.

#Key Features

• Joint segmentation and classification: masks, boxes and cell-type labels come out of one forward pass and one shared encoder, rather than from a segmenter feeding a downstream classifier.
• Multiscale objects: cells, nuclei and much larger non-cellular structures are detected by the same model, which conventional cell segmenters cannot represent.
• Calibrated confidence: every detection carries a confidence score, and that score tracks classification accuracy closely (regression coefficient 0.56, against 0.21 for CellSighter and 0.19 for Mask R-CNN).
• Modality breadth: multiplexed fluorescence, brightfield and non-microscopy images all work, as do imaging-based spatial transcriptomics inputs where DAPI and transcript signals are stacked as channels.
• Released checkpoints: the pip-installable cellotype package runs predict() on a project-supplied checkpoint, with no training step on the user's own data.

#Technical Details

The backbone is a Swin-L Transformer initialized from COCO instance-segmentation weights, as are the DINO and MaskDINO parameters; training uses Adam at a learning rate of 10⁻⁶ and batch size 8, with 1,000 object queries at inference. On TissueNet — 2,580 training patches of 512 × 512 pixels and 1,324 test patches spanning six imaging platforms (CODEX, CyCIF, IMC, MIBI, MxIF, Vectra) and six tissue types — the confidence-score variant reached a mean average precision of 0.56 for cell segmentation and 0.66 for nuclear segmentation over 10-fold cross-validation, against 0.35 and 0.52 for Cellpose2 and 0.31 and 0.24 for Mesmer. On the Cellpose Cyto dataset it reached 0.47 versus 0.32 for Cellpose2, and on Xenium data with DAPI plus transcript input 0.47 where the transcript-based methods SCS and Baysor scored 0.01. A colorectal cancer CODEX dataset of 58 channels and 11 annotated cell types gave a mean AP of 0.55 for joint segmentation and classification, against 0.13 for a Cellpose2 plus CellSighter pipeline and 0.43 for Mask R-CNN; a 54-channel bone marrow dataset gave 0.39 for adipocytes, 0.31 for trabecular bone fragments and 0.42 for the remaining cell types.

#Applications

The immediate use is automated annotation of image-based spatial omics experiments — producing the cell boundaries and cell-type labels that downstream neighborhood and differential-abundance analyses depend on, without the manual gating step that normally follows segmentation. Tissue atlas efforts working through large multiplexed CODEX, CyCIF, IMC or MIBI cohorts benefit most, as do spatial transcriptomics studies where membrane stains are unavailable and segmentation must lean on DAPI and transcript density together. The multiscale behavior suits tissues whose readout depends on non-cellular architecture, such as bone marrow trabeculae.

#Impact

CelloType demonstrated that transformer detection architectures developed for natural images transfer to multiplexed tissue images with many more channels than RGB, and that treating segmentation and classification as one multitask problem improves both. The code is Apache 2.0 licensed, checkpoints and documentation are released, and the paper is open access. The main constraint is on the classification side: the classifier is trained against a specific antibody panel and cell-type taxonomy, so a pretrained classification model does not transfer to a new panel unless the cell and structure types substantially overlap — a limit the authors state directly. The segmentation leg is the portable one, and the authors point to few-shot and contrastive learning as the route to generalizing classification.