LSP-DETR

A microtome section is thinner than the nuclei it cuts through, so nuclei that sat above one another in the tissue project onto outlines that genuinely intersect on the slide. Almost every nuclei segmentation pipeline forbids exactly that: the network emits a per-pixel mask, and a hand-written separation step — a watershed transform, or non-maximum suppression over candidate polygons — carves it into disjoint instances. That step is also where much of the compute goes, running on the CPU and scaling worse than linearly in image area, so on a gigapixel slide it, not the network, sets the throughput.

LSP-DETR, built by informaticians at Masaryk University with pathologists at Masaryk Memorial Cancer Institute in Brno, removes the mask and the separation step together. Each nucleus is a star-convex polygon — a centre point plus 64 radial distances along evenly spaced rays — and the model is a DETR-style set predictor: object queries laid out on a grid over the image, initialised as circles and iteratively morphed into the outline of whichever nucleus sits in their cell. Queries that find nothing classify themselves as "no nucleus" and drop out, so the deduplication NMS would perform is learned rather than imposed.

The polygon representation comes from StarDist, which predicts radial distances per pixel and reconciles redundant candidates with NMS — a step that structurally cannot return two overlapping outlines. LSP-DETR's radial-distance loss removes the constraint: rather than demanding an exact boundary, it bounds each ray between the nearest edge of any ground-truth nucleus containing the predicted centre and the nearest pixel covered by no nucleus, penalising only predictions outside that band. Overlaps emerge even though the training annotations are strictly non-overlapping.

#Key Features

• Fully end-to-end: The final set of nuclei comes off the GPU directly, with no CPU-bound pre- or post-processing stage.
• Polygons as the output primitive: 64 radial distances from a learned centre describe each nucleus, replacing the per-pixel mask with a compact shape descriptor for downstream morphometry.
• Overlap-tolerant radial loss: Bounding each ray between a minimum and maximum plausible distance lets realistic nuclear overlaps appear without explicit annotation.
• Linear-complexity attention: Sliding Tile Attention restricts attention to local neighbourhoods, giving O(n) cost in image area and allowing inference on images up to 8192 × 8192 pixels on one H100 GPU.
• Resolution-aware query grid: Grid spacing derives from the physical pixel size at a fixed 3.5 µm cell diameter, so one checkpoint covers any magnification.

#Technical Details

A Swin-V2-T backbone, fine-tuned from ImageNet-1K weights, produces three multi-scale feature maps fed round-robin into a six-layer transformer decoder; rotary positional encodings supply relative geometry to both the self- and cross-attention paths. Training follows the CellViT recipe — 130 epochs, AdamW, batch size 16, backbone frozen for the first 30 — on PanNuke: 7,901 images of 256 × 256 pixels at ×40 magnification (0.25 µm/px), 189,744 annotated nuclei, 19 tissue types and five nucleus classes. The model has 45M parameters and 26G FLOPs, against 163.8M for LKCell and 699.7M for CellViT.

On PanNuke three-fold cross-validation it reaches 67.5 bPQ and 48.2 mPQ, a little behind LKCell, while posting the highest detection F1 in all five nucleus categories. Because panoptic quality forbids overlapping predictions, those scores require a watershed refinement applied purely for evaluation; under an overlap-insensitive variant of the metric it ranks first on binary mask quality, at 68.8 bMPQ. Transferred unchanged to MoNuSeg — full 1000 × 1000 pixel images in a single pass despite 256 × 256 training — it reaches F1 85.0 against LKCell's 83.0. Predicted polygon overlap agrees with consensus annotations from two expert pathologists at F1 0.964, and inference on TCGA colorectal slides runs at 0.45 s/mm², a 3.2× speedup over StarDist.

#Applications

The model targets slide-scale quantification: cell counts and density maps, nuclear morphometry for grading, tumour-infiltrating lymphocyte scoring, and cell-graph analysis of the tumour microenvironment — all of which need every nucleus on a slide, not a curated region of interest. Being input-size agnostic and needing no tiling with overlapping margins, it slots into a laboratory pipeline as a single GPU stage turning a scanned slide into a set of polygons.

#Impact

LSP-DETR shows that the mask-plus-separation convention carries two costs — throughput and the inability to represent overlap — and that set prediction over shape descriptors pays neither. It sits alongside CellNuc-DETR, which reaches for the same DETR framing but abandons the outline in favour of detection. The limits are real: no public dataset annotates overlapping nuclei, so the central claim can only be assessed qualitatively by a pathologist; accuracy on standard PanNuke metrics trails the much larger LKCell; and the released checkpoint was trained without class labels, so it localises nuclei but does not type them. Code is MIT-licensed, the weights are on HuggingFace under Apache 2.0, and the work is published in Computer Methods and Programs in Biomedicine.