CircleFormer

A 512×512 patch of H&E tissue can hold close to a thousand nuclei, and many of them touch. A heatmap detector predicts a peak per object, then deletes near-duplicate peaks with non-maximum suppression — exactly the wrong instinct when two nuclei genuinely overlap, because the second true detection looks like a duplicate of the first. CircleFormer removes that post-processing step rather than tuning it: detection is posed as set prediction, a Hungarian matcher assigns each nucleus to at most one query, and nothing is suppressed afterwards.

What it keeps from CircleNet is the target. A nucleus is approximately round, so it is described by a center and one radius rather than a center and two side lengths, and that description does not change when the slide is scanned at a different angle. CircleFormer carries the circle into a DETR-style decoder: following DAB-DETR, each object query is a dynamic anchor circle (x, y, r), and each decoder layer predicts an increment (Δx, Δy, Δr) that refines it. The radius is not just an output — it feeds back into attention. Circle-modulated attention rescales the positional attention map by the ratio of a predicted reference radius to the anchor radius, and deformable circle cross attention draws its sampling offsets as a radius and an angle in polar coordinates, so every sampled key lands inside the query's own circle rather than in a rectangle around it.

The model was built by Erkang Cheng's group at the Institute of Intelligent Machines of the Chinese Academy of Sciences, with collaborators at Anhui University and Zhengzhou University, and presented at MICCAI 2023.

#Key Features

• Anchor circle as the object query: Three degrees of freedom rather than four, refined layer by layer through the decoder stack instead of predicted once.
• Radius-modulated cross attention: The query's own radius scales the positional attention map, pooling features at a scale matched to the object the query is chasing.
• Deformable sampling inside the circle: Offsets are parameterized as (Δr, Δθ) and converted to Cartesian coordinates; uniform initialization of the reference points beats random placement.
• Generalized circle IoU as a training loss: Two discs are scored against the smallest circle enclosing both, which yields a gradient when they are disjoint. CircleNet used circle IoU only for evaluation; here it drives regression.
• Optional mask branch: A feed-forward head turns each decoder embedding into a 28×28 mask under dice and cross-entropy losses, so one model does detection and instance segmentation.

#Technical Details

All variants use a ResNet-50 backbone with six transformer encoder and six decoder layers, 8 attention heads, 4 reference points per cross-attention module, and 1000 object queries — chosen because the densest training image approaches a thousand nuclei — initialized from a COCO-pretrained detector. Training and evaluation use MoNuSeg 2018, the Multi-Organ Nuclei Segmentation challenge set drawn from TCGA slides: 30 training and validation images plus 14 test images of lung and brain tissue, from which ten random 512×512 patches per image give 200 training, 100 validation and 140 testing patches.

On nuclei detection, CircleFormer reaches 49.7 AP, the deformable variant CircleFormer-D 52.9, and the joint detection-and-segmentation variant CircleFormer-D-Joint 53.0 AP with 90.0 AP₅₀ and 59.0 AP₇₅ — against 48.7 for CircleNet with an Hourglass backbone, 49.6 for DAB-Deformable-DETR, 41.6 for Faster R-CNN and 22.6 for plain DETR. Joint detection and segmentation reports 44.4 mask AP against 41.3 for SOIT and 38.0 for QueryInst. Ablations isolate each piece: circle-modulated attention lifts AP from 45.7 to 48.6, and swapping circle IoU for generalized circle IoU adds a further 2.2 points. The circle costs accuracy on medium-sized objects, where AP falls to 32.8 while a box-based CenterNet reaches 39.5 — those are the elongated nuclei a single radius cannot describe.

#Applications

Nuclear counts, density and morphology underpin tumor grading and much of quantitative computational pathology, all of it downstream of finding every nucleus in a field of view. A detector that emits a center, a radius and a mask in one pass gives those measurements directly, and three numbers per nucleus are compact enough to store slide-wide.

#Impact

CircleFormer carried circle representation from CNN detectors into the DETR family, and its generalized circle IoU turned a circle overlap measure from an evaluation metric into a trainable regression loss. The catalog holds neighbours on both sides — CircleNet and CircleSnake pursue circles with convolutional backbones, while CellNuc-DETR later applied deformable set prediction to nuclei with ordinary boxes. The evidence is narrow: one dataset, one object class, no external validation cohort, no reported runtime, and experiments on 512×512 patches rather than the whole slides of the title. The release is narrow too — the repository carries model-definition modules under the Apache 2.0 license, but no trained checkpoint, no training or inference script and no configuration files, so reproducing the reported numbers means rebuilding the surrounding pipeline.