Panoptic segmentation network for breast cancer H&E slides that maps tissue regions and nuclei to score tumor-infiltrating lymphocytes.
No providers recorded yet. Browse all providers
International guidelines score stromal tumor-infiltrating lymphocytes (TILs) in breast cancer as the share of stroma occupied by lymphocytes, which means a TILs score needs two answers from the same slide: where the stroma is, and which cells inside it are lymphocytes. Most deep-learning tools answer only one. A tissue segmenter and a separate nucleus classifier can disagree — a lymphocyte called inside a tumor nest, a fibroblast called in epithelium — and the pipeline then has to reconcile predictions that were never made with each other in view.
MuTILs answers both questions in one network. Developed in Lee Cooper and Mohamed Amgad's pathology data science group at Northwestern University with breast pathologist Roberto Salgado (GZA-ZNA hospitals, now Ziekenhuis aan de Stroom, Antwerp, and Peter MacCallum Cancer Centre), it was first posted to medRxiv in January 2022 and published in npj Breast Cancer in June 2024. It is a panoptic segmentation model: semantic segmentation for tissue regions and instance segmentation for nuclei, with the tissue answer used to constrain the cell answer.
MuTILs is also a concept bottleneck model. Rather than predicting a TILs score or patient outcome directly from pixels, it predicts the components a pathologist counts — stromal regions, TIL nuclei, all nuclei — and the score is computed from those interpretable quantities. Its dual-resolution panoptic design was later extended to colorectal cancer by an unrelated group as CAMuTILS.
MuTILs comprises two parallel U-Nets of depth 5, trained with a multi-task loss that weights region predictions for the region of interest and the high-power field equally. Training used PanopTILs, a dataset released with the paper that reconciles the group's earlier BCSS tissue-region and NuCLS nucleus annotations over TCGA invasive breast carcinomas: 1,709 regions of interest from 151 patients containing 814,886 nuclei. Manual nucleus labels cover the central 256×256 pixels of each 1024×1024 ROI, and the periphery was filled with labels extrapolated by earlier nucleus models; 85 additional slides from the Cancer Prevention Study II cohort supplemented training. Evaluation used 5-fold internal-external cross-validation that holds out whole hospitals, and at inference the five fold models are ensembled by assigning ROIs to them in rotation.
On held-out hospitals, MuTILs reached an AUROC of 0.93 for lymphocyte nuclei and a DICE of 0.81 for stroma. The region constraint raised nucleus classification AUROC by about 2–3%, mainly by stopping immature fibroblasts and large TILs from being called cancer. The model does not separate normal from neoplastic epithelium at 10x, so those classes are merged for scoring. On 305 TCGA breast carcinoma patients, computational scores correlated with two pathologists' visual scores at Spearman R 0.55–0.61.
MuTILs is built for breast cancer researchers and computational pathologists who need reproducible stromal TILs scores aligned with the International Immuno-oncology Biomarker Working Group recommendations, for example in retrospective cohort studies of prognosis or treatment response. Its per-slide outputs — nucleus metadata and morphometric features, ROI masks, and a combined whole-slide segmentation — also support broader mapping of the breast tumor microenvironment. The released Docker image runs inference on whole-slide images directly.
MuTILs showed that a computational TILs score built from interpretable components can carry more prognostic signal than visual scoring. In TCGA infiltrating ductal carcinomas the nTnS score was the only metric independently prognostic for progression-free interval after adjusting for age and stage, and in Her2+ carcinomas all computational scores were independently prognostic while the visual score was not. The validation used TCGA alone, and the authors flag multi-cohort validation as future work. PanopTILs, released openly alongside the model, gives other groups a joint region-and-nucleus training resource, and CAMuTILS has already reused the architecture in colorectal cancer.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.