Cell segmentation for image-based spatial transcriptomics that fuses RNA point clouds with any number of membrane and nuclear staining channels.
No providers recorded yet. Browse all providers
Imaging-based spatial transcriptomics resolves individual RNA molecules in tissue, but every downstream result — the count matrix, the cell type calls, the ligand-receptor inference — depends on one upstream decision: which cell does each transcript belong to. A segmentation algorithm makes that call, and in practice it is the weakest link. Image-based segmenters such as Cellpose rely on membrane stains that bind cell-type-specific proteins and so leave stretches of tissue effectively unlabelled, while point-cloud methods working directly on transcripts struggle to separate neighbouring cells with similar expression profiles.
RNA2seg, from Thomas Defard, Alice Blondel and colleagues at Institut Pasteur and Mines Paris PSL, takes both inputs at once. It consumes an arbitrary number of nuclear and membrane staining channels together with the RNA point cloud, so it uses high-quality staining wherever it exists and falls back on transcript density wherever it does not. The design intent is a generalist: one checkpoint applied across tissues, gene panels, and assay chemistries rather than a model refit per dataset.
The bottleneck for training such a model is annotation: manually tracing cell boundaries in spatial transcriptomics is prohibitively slow, and no large annotated corpus exists. The authors therefore trained RNA2seg with a teacher-student scheme in which Cellpose supplies segmentations only where staining is good, consistency rules discard those disagreeing with the nuclear segmentation, and unlabelled pixels are excluded from the gradient — removing the annotation ceiling and exposing the model to over 4 million cells.
The released RNA2seg is a CNN with 6.6 million parameters, trained Cellpose-style to predict gradient flow fields toward cell centres; a Mamba-based variant at 44.2 million parameters performed comparably, so the smaller CNN was kept. Training used over 4 million cells across seven human organs — breast, colon, lung, melanoma, uterine, prostate and ovarian — from two chemistries, Vizgen MERFISH and CosMx, with DAPI, poly-T and three cell-boundary channels. Evaluation rests on 724 manually annotated cells across five datasets, labelled with a co-expression colouring scheme that projects spatially correlated genes into RGB so annotators can read boundaries out of the point cloud.
On withheld regions of the training datasets, RNA2seg matched or beat staining-based and point-cloud baselines, with the largest margins where staining is poor: in a breast example it reached IoU 0.71 against 0.52 for Cellpose and 0.48 for the vendor segmentation. Zero-shot transfer was tested on a mouse ileum MERFISH dataset differing in species, tissue, gene panel and staining scheme — one cell boundary channel, no poly-T — where RNA2seg outperformed ComSeg, Baysor and a Cellpose model its authors had fine-tuned on that very dataset. It also produced markedly fewer spurious cells, those containing no nucleus, than any baseline. Reported scores average ten independently trained models.
RNA2seg targets anyone running MERFISH, CosMx or comparable assays who needs cell boundaries before single-cell analysis can begin, and it is most valuable where membrane markers are heterogeneous — tumour sections, fibrous stroma, densely packed epithelium. It installs as a pip package built on SpatialData and SOPA and reads OME-Zarr, so it drops into existing large-scale pipelines. Labs working far from the training distribution can run the automatic fine-tuning path instead of commissioning annotations.
RNA2seg makes a concrete argument about where segmentation information lives in spatial transcriptomics: transcript positions alone carry enough signal to rescue regions that defeat staining-based methods, and the gain survives transfer to unseen species and panels. The ablation also reports a negative result worth knowing — adding gene identity to the RNA image, through co-expression colouring or learned per-gene colours, did not improve accuracy over positions alone. The manually annotated benchmark is released publicly and fills a real gap, since the field has largely validated segmentation through indirect proxies. The model is restricted to 2D sections, limiting it for thick samples and organoids. The work was peer reviewed and published in Genome Biology in January 2026 under CC BY 4.0; the earlier bioRxiv preprint carries the more restrictive CC BY-ND terms, so the version of record is the one to reuse from.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.