DeepCell Types

Spatial proteomics measures dozens of proteins in intact tissue, but turning those multiplexed images into a map of cell types is a persistent bottleneck. Each experiment chooses its own antibody panel, and different platforms — MIBI, CODEX, imaging mass cytometry, and others — capture different markers at different resolutions. A classifier trained on one panel typically fails on another, forcing researchers to hand-gate cells or retrain models for every dataset.

DeepCell Types, developed by the Van Valen Lab at the California Institute of Technology, addresses this by treating the marker panel as language rather than a fixed set of input channels. It is a language-informed vision model: a transformer with channel-wise attention pairs a visual encoder over per-cell image crops with a language encoder over the names and semantics of each marker. Because the model reasons about what a channel means rather than which position it occupies, it classifies cell types zero-shot on panels and platforms it never saw during training, matching each image's channels against a shared marker and cell-type registry.

Posted to bioRxiv in November 2024, DeepCell Types extends the Van Valen Lab's DeepCell ecosystem — which includes the CellSAM segmentation model — from finding cells to identifying them, providing the phenotyping step that follows segmentation in a spatial-proteomics pipeline.

#Key Features

• Language-informed phenotyping: A language encoder embeds each marker's name and meaning, letting the model interpret arbitrary antibody panels instead of assuming a fixed channel order.
• Channel-wise attention: The transformer attends across image channels so that the presence, absence, or combination of markers on a given cell drives its predicted identity.
• Zero-shot generalization: New datasets are handled by matching their channels against a marker and cell-type vocabulary, so no retraining is required for unseen panels, tissues, or imaging platforms.
• Trained on Expanded TissueNet: A diverse corpus of 9.8 million annotated cells spanning nine imaging platforms grounds the model across tissue and modality diversity.
• Released weights and inference package: A pretrained checkpoint is distributed through free registration, and a pip-installable Python package exposes a single predict() call over raw images, segmentation masks, channel names, and pixel size.

#Technical Details

DeepCell Types combines a visual encoder that operates on segmented single-cell crops with a language encoder that embeds marker semantics, fused through a transformer with channel-wise attention and topped by a residual-MLP classification head. A packaged vocabulary file maps image channels to markers and cell types, so channel-to-marker matching happens without external lookups at inference time. The model is trained in two stages: backbone training with weighted sampling to balance rare cell types, followed by head-only retraining. Training uses Expanded TissueNet, which combines the original TissueNet data with HuBMAP-derived annotations to reach 9.8 million cells across nine imaging platforms. The paper reports robust phenotyping across cell types, tissues, and imaging modalities, including generalization to marker panels and platforms outside the training distribution. Code is released under a Modified Apache 2.0 license with an academic and non-commercial carve-out, and pretrained weights download via a free account at users.deepcell.org.

#Applications

DeepCell Types serves researchers building single-cell maps of tissue from multiplexed imaging, including tumor microenvironment studies, immunology, developmental biology, and large tissue-atlas efforts such as HuBMAP. Because it accepts whatever panel an experiment used, it removes the need to hand-design a classifier or manually gate populations for each new study, making consistent cell-type calls across cohorts and platforms feasible. It slots naturally downstream of a segmentation model, taking raw images and masks and returning per-cell identities that feed spatial neighborhood and cell-cell interaction analyses.

#Impact

By decoupling cell phenotyping from any specific marker panel, DeepCell Types offers a route to standardized, comparable cell-type annotation across the fragmented landscape of spatial proteomics, a field where panel and platform differences have long blocked reuse of trained models. The work is a preprint, and its released weights sit behind a free registration step and a non-commercial license, so unrestricted redistribution is limited; benchmark comparisons are reported as superior zero-shot phenotyping rather than through a single headline metric. Its integration into the widely used DeepCell tooling positions it for adoption by labs already relying on that ecosystem for segmentation.