CellNuc-DETR

Counting and typing cell nuclei is the foundation of quantitative digital pathology: it underpins cell population estimates, biomarker quantification, and the cell-graph analyses used to characterise tumour microenvironments. The established approach is instance segmentation — CellViT, HoVer-Net, and HoVer-NeXt all predict a mask per nucleus and then run watershed-style post-processing to separate touching instances. That post-processing is the bottleneck. A whole slide image can cover hundreds of square millimetres and contain millions of nuclei, and mask generation plus instance separation makes slide-level inference slow enough to obstruct routine clinical use.

CellNuc-DETR, from Oscar Pina and Verónica Vilaplana at the Universitat Politècnica de Catalunya with Eduard Dorca at Hospital Universitari de Bellvitge, argues that the mask is usually not what the pathologist needs. What matters downstream is where each nucleus is and what type it is. Reframing the problem as object detection removes the mask head and its post-processing entirely, and lets the model be built on the DETR family of set-prediction detectors, where each object query directly emits a box and a class label.

The result is a single checkpoint trained on PanNuke that both improves accuracy on held-out data and generalises to datasets it never saw during training, while running fast enough to process full slides.

#Key Features

• Detection instead of segmentation: Object queries emit nuclear location and class directly, eliminating mask decoding and instance-separation post-processing.
• Multi-scale deformable attention: A hierarchical Swin Transformer backbone feeds a deformable-DETR encoder-decoder, so attention samples a sparse set of reference points per query rather than the whole feature map — important when a tile holds hundreds of small, densely packed nuclei.
• Cross-dataset generalisation: The PanNuke-trained checkpoint is applied to CoNSeP and MoNuSeg with no fine-tuning, competing against methods trained directly on those datasets.
• Whole-slide inference pipeline: A sliding-window scheme partitions feature maps rather than images, aggregating context across large tiles for practical full-slide processing.
• Released checkpoints at two magnifications: Trained models are provided for 0.25 µm/px and 0.50 µm/px with Swin-T and Swin-L backbones, letting users trade accuracy against throughput.

#Technical Details

The architecture is a two-stage deformable DETR over a hierarchical backbone that produces a multi-level feature pyramid; the encoder refines these features with multi-scale deformable self-attention and the decoder maps object queries to bounding boxes and class labels. Backbone variants evaluated include ResNet-50 and Swin-Tiny, Base, and Large, with Swin backbones consistently ahead of ResNet-50; the number of pyramid levels fed to the transformer had little effect. Models were trained for 100 epochs, using the PanNuke three-fold cross-validation split for benchmarking and an 80/20 split for the released checkpoints.

PanNuke supplies 7,904 patches of 256 × 256 pixels at 0.25 µm/px across 19 tissue types and five nucleus classes. Evaluation follows the HoVer-Net protocol: bipartite matching of predicted to ground-truth centroids within a 12-pixel radius, then detection and per-class F1. The three released checkpoints reach detection F1 of 82.67 (Swin-T, 0.25 µm/px), 81.77 (Swin-T, 0.50 µm/px), and 83.06 (Swin-L, 0.25 µm/px) on the held-out PanNuke fold. Cross-dataset transfer to CoNSeP — with PanNuke's five classes mapped onto CoNSeP's four — and detection-only evaluation on MoNuSeg both exceed methods trained on those datasets directly. On 20 TCGA slides averaging 300 mm² each, the pipeline runs roughly twice as fast as HoVer-NeXt and about ten times faster than CellViT-256.

#Applications

The model targets pathology workflows that need cell-level statistics over entire slides rather than curated regions of interest: tumour-infiltrating lymphocyte scoring, cellularity and biomarker quantification, and spatial or cell-graph analysis of the tumour microenvironment. Because it generalises across tissue types and staining conditions without retraining, it can be dropped into a laboratory's slide pipeline as a preprocessing step producing a typed point cloud of nuclei, which downstream spatial statistics consume directly.

#Impact

CellNuc-DETR makes a concrete case that the segmentation-first convention in computational pathology carries a cost that most clinical applications do not need to pay, and that a detection formulation recovers both accuracy and an order of magnitude in slide-level throughput. It does not produce nuclear morphology features — area, eccentricity, contour — so studies that depend on shape descriptors still require a segmentation model. The work is an arXiv preprint; code and Google Drive-hosted weights are public and the repository's README declares an MIT license, though no license file is present in the repository itself.