Cellpose

Overview

Cellpose is a generalist algorithm for cell and nucleus instance segmentation in microscopy images, developed by Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu at HHMI Janelia Research Campus. Published in Nature Methods in December 2020, it introduced a fundamentally different approach to instance segmentation: instead of predicting bounding boxes or pixel class labels directly, Cellpose predicts a spatial flow field derived from simulated heat diffusion, then integrates those flows to recover cell instances.

The core challenge the paper addresses is generalization across imaging contexts. Prior segmentation methods — including Mask R-CNN and watershed-based approaches — required per-dataset retraining or hand-tuned parameters to handle the variety of cell sizes, shapes, and imaging modalities encountered in biological research. Cellpose solves this by representing every interior pixel as a gradient vector pointing toward the cell's centroid. This representation is invariant to cell size and shape, and pairs with a style vector that automatically estimates cell diameter from the image, eliminating manual scale setting for most imaging setups.

Cellpose has since become one of the most widely deployed cell segmentation tools in biological imaging, and has been extended through Cellpose 2.0 (2022) and Cellpose3 (2025). This entry covers the original 2020 publication.

Key Features

• Flow-based instance segmentation: Predicts a simulated diffusion gradient field rather than pixel labels or bounding boxes, enabling robust separation of touching and overlapping cells regardless of their size or shape.
• Automatic diameter estimation: A 64-dimensional style vector computed from the deepest encoder features drives a linear regression model that predicts cell diameter, removing the need for manual scale configuration.
• Generalist training: A single model trained across 608 images spanning cell lines, tissue sections, fluorescence, brightfield, and phase contrast handles diverse imaging setups without retraining.
• Zero-shot 3D segmentation: Extends 2D predictions to volumetric data by applying the model slice-by-slice across z-stacks, without requiring any labeled 3D training images.
• Nucleus model: A second pretrained model targets DAPI-stained nuclear channels and can be combined with the cytoplasm model for paired cell-nucleus segmentation.
• Python API and GUI: pip-installable package with GPU support, plus a graphical interface for annotation and batch processing without writing code.

Technical Details

Cellpose uses a modified U-Net architecture. The encoder follows a standard convolutional downsampling path; the decoder upsamples with skip connections from matching encoder levels, combining features additively rather than through concatenation. A global average pooling operation over the deepest encoder feature maps produces a 64-dimensional style vector that is broadcast and added to each spatial position in the decoder, providing whole-image context at every scale. The same style vector feeds a separate linear regression head that predicts cell diameter for automatic scale normalization.

The network produces three output channels: horizontal flow, vertical flow, and a binary cell probability map. Flow targets are computed during training by simulating heat diffusion from each mask centroid outward, producing a smooth gradient field in which every interior pixel points toward its cell's center. At inference, pixels above a probability threshold are integrated along the predicted flow vectors; pixels that converge to the same point are grouped into candidate instances, and small isolated instances are discarded.

The training set comprised 608 images containing more than 70,000 individually segmented objects drawn from mammalian cell lines, tissue sections, bacteria, fluorescence, brightfield, and phase contrast sources. On held-out test images, Cellpose achieved an average precision (AP) of 0.786 at IoU 0.5, outperforming Mask R-CNN (0.359), StarDist (0.553), and UNet++ (0.580). The model also exceeded 3D-specific methods on confocal z-stacks despite using only 2D supervision.

Applications

Cellpose serves as a plug-and-play segmentation step in imaging pipelines built around CellProfiler, FIJI/ImageJ, and napari, with plugins available for each environment. In high-content screening, it handles the diversity of cell lines and imaging conditions across multi-well plates without per-plate parameter tuning, with GPU-accelerated batch inference supporting processing of thousands of images. The flow representation handles touching cells in tissue sections more reliably than threshold-based watershed methods. The pretrained weights also serve as a fine-tuning starting point for specialized cell types — such as neurons, cardiomyocytes, or organoids — where a modest number of additional annotated images can substantially improve performance.

Impact

Cellpose became the dominant general-purpose cell segmentation tool in biological imaging within a few years of publication, with the Nature Methods paper accumulating thousands of citations. Its generalist training strategy and flow-based representation influenced subsequent segmentation work, and the model's plug-in ecosystem lowered the barrier for wet-lab researchers without computational expertise to apply deep learning to their imaging data. Notable limitations include reduced performance on densely packed cells sharing low-contrast boundaries, fragmented masks on highly elongated cells such as neurons and myofibers, and artifacts from the slice-by-slice 3D approach when voxel spacing is anisotropic. The original training set was not publicly released, limiting independent auditing of the base training distribution. These limitations motivated the development of Cellpose 2.0, which added human-in-the-loop retraining capabilities.