CL-Net

Computed tomography is the most widely used cross-sectional imaging modality in radiology and oncology, yet the deep-learning models that parse it remain narrow. Each is typically trained on a single dataset covering a handful of structures, because no CT collection annotates the whole body: manually delineating hundreds of anatomies in 3D is prohibitively expensive and demands specialist expertise. CL-Net, developed at DAMO Academy with a clinical consortium spanning China, Taiwan, Singapore, and the United States, treats that fragmentation as the problem to solve rather than a constraint to work around.

CL-Net is a single unified network that segments 235 fine-grained whole-body anatomies from a 3D CT volume: 193 organs, 33 lymph node stations, and 9 lesion types, including 73 structures that no public segmentation dataset annotates. It was assembled from 36 separately labeled datasets, none of which covers more than a fraction of that list. Instead of forcing one output head to learn every class at once, CL-Net trains a general encoder, freezes it, then attaches a small dedicated decoder per organ or organ group. Adding an anatomy means adding a decoder, so nothing already learned can be overwritten.

That design sets CL-Net against the "segment anything"-style medical foundation models that dominate recent work. On anatomies the competitors were trained to handle, it leads VISTA3D by 2.9% Dice and SAT-Pro by 9.9%, and it exceeds the 2D prompt-driven BiomedParse by 67.0% Dice when both are scored volumetrically. A preliminary version of the method appeared at ICCV 2023; this expanded work is a preprint and has not yet been peer reviewed.

#Key Features

• Clinically comprehensive coverage: One model spans head and neck, chest, abdomen, pelvis, and the skeleton, including fine-grained vessels, muscle substructures, glands, and lymph node stations that broader segmentation tools omit.
• Frozen general encoder: The shared encoder is trained once and then held fixed, so decoders added later inherit a stable representation and catastrophic forgetting becomes structurally impossible rather than merely regularized against.
• Lottery-ticket decoder pruning: Each decoder is iteratively pruned to roughly 1–3% of a full-size decoder, keeping the complete 235-class model small enough to run on modest hardware.
• Body-part-guided merging: A body part regression module scores each axial slice by relative anatomical height, so decoder outputs are merged within their expected body region and distal false positives are suppressed.
• Two training regimes: The same architecture supports universal segmentation, where all datasets are available at once, and continual segmentation, where datasets arrive one at a time and earlier data cannot be revisited.

#Technical Details

CL-Net is a convolutional encoder-decoder built on the nnU-Net v1 framework. Its general encoder is trained on the public TotalSegmentator V2 dataset; the authors report that adding further datasets or self-supervised pretraining did not improve it. Development drew on 13,952 CT scans from 36 partially labeled datasets, 20 public (4,855 scans) and 16 private (9,097 scans), across multiple vendors, contrast phases, and pathologies. Across all 235 anatomies CL-Net reaches a mean Dice of 86.1% and a mean average surface distance of 1.17 mm, versus 83.9% and 1.32 mm for an ensemble of 36 dataset-specific nnU-Nets, while occupying 55.5 MB of parameters against the ensemble's 1,126.8 MB, roughly a twentieth of the size. Gains concentrate on small, low-contrast structures: pituitary (+10.1% Dice), optic nerves (+11.8%), chiasm (+11.9%), hypothalamus (+13.9%), and brachial plexus (+13.8%). On seven held-out external datasets covering 1,979 scans and 43 anatomy types, it stays 1.3% Dice ahead of the ensemble with 14.6% lower surface distance error. Trained sequentially over five datasets and averaged across four dataset orderings, it reaches 92.6% mean Dice, where the continual-learning baselines MiB, PLOP, and CSCLIP collapse to 30.0%, 46.0%, and 43.6%.

#Applications

Fine-grained whole-body parsing is the input layer for a range of quantitative clinical work: contouring organs at risk and lymph node stations for radiotherapy planning, quantifying tumor burden and tracking it across follow-up scans, and opportunistic screening that mines body composition and cardiometabolic measurements from CT ordered for unrelated indications. Radiation oncologists and radiologists benefit most directly, and because each anatomy owns an independent decoder, a site can fine-tune one structure on local data without retraining or revalidating the rest of the model.

#Impact

CL-Net is a pointed empirical argument that prompt-driven, SAM-derived architectures have not yet displaced task-specific convolutional segmentation in 3D medical imaging: the paper shows leading medical foundation models trailing plain nnU-Net baselines on volumetric CT, and attributes the gap to the difficulty of resolving low-contrast anatomical boundaries with long-range attention. Code is public on GitHub and a 235-class checkpoint is published on Hugging Face under Apache-2.0, though the paper describes the release as being for non-commercial use. Sixteen of the 36 training datasets are private, so the exact training corpus cannot be reconstructed by outside groups.