SEERFound

Mesothelioma, salivary gland cancer and paediatric neuroblastoma each pose a well-defined prognosis question and supply almost nothing to answer it with. Rare cancers make up roughly 20% of diagnoses in the United States and 71% of cancers in children and adolescents, yet an individual rare-cancer cohort seldom carries enough deaths for a multivariable time-to-event model. Oncology's convention is local — one model per cancer, fitted from that cancer's own labelled outcomes — so the diseases most in need of a prognostic model are the ones least able to supply the labels to fit one.

SEERFound, built at Sir Run Run Shaw Hospital of Zhejiang University School of Medicine, asks whether a representation learned with no outcome labels at all can stand in for those missing labels. It is a Transformer encoder pretrained by masked field-value modelling on 9,425,135 tumour records from the SEER 17 registries diagnosed between 2000 and 2023, reading only diagnosis-time facts: primary site, histology, grade, summary stage, tumour size, age. Pretraining produced a single terminal checkpoint at optimizer step 11,084, used unchanged downstream; the encoder is then frozen in evaluation mode and read by a zero-initialised 256-parameter linear Cox head fitted separately within each cancer.

Structured-clinical-record encoders such as CEHR-BERT and Clin-JEPA pretrain on longitudinal hospital records and test transfer inside one source population. SEERFound's substrate is the coded population registry, and its transfer test is posed at the level of the disease: nine rare cancers were removed from the pretraining corpus outright, each requiring its own pretraining run, so the cancer being scored contributed nothing to the representation scoring it.

#Key Features

• Outcome-agnostic pretraining: random masking of 15% of non-CLS fields alternates with semantic-block masking of stage, metastasis, grade, size or morphology, and per-field softmax heads predict only values legal for the masked field; outcome and treatment fields are blocked from the vocabulary.
• Dual-view era harmonisation: each record is emitted twice, as an era-specific raw view of 22 field slots and a harmonised canonical view of 18, each plus CLS, so two decades of registry recoding — notably the 2018 move to AJCC 8th-edition staging — stays visible rather than silently collapsed.
• Fail-closed coding gate: a frozen specification over all 250 fields of the SEER Research Plus extract admitted 23 diagnosis-time inputs; AJCC and EOD T, N and M categories, site-specific factors and biomarkers were excluded as era-inconsistent.
• Leave-cancer-out transfer test: nine rare cancers, from acute myeloid leukaemia to paediatric neuroblastoma, were each held out of a pretraining run with the vocabulary refitted, so the evaluation is leakage-free.

#Technical Details

The encoder is an 8-layer, 8-head pre-norm Transformer of width 256 with no positional encoding; non-CLS tokens are shuffled, and each token sums field-identity, value and missing-state embeddings. Pretraining ran three epochs on a single RTX 6000D in 2.94 hours, over 15,079,716 training sequences from patient groups partitioned 80/10/10 with verified zero crossover. On the sealed test partition, all nine leave-cancer-out models exceeded an architecture-identical random frozen encoder in Harrell concordance, from +0.0034 in chronic lymphocytic leukaemia to +0.0368 in paediatric neuroblastoma, every lower 95% limit above zero. At a budget of 256 labelled patients, all 67 cancer targets favoured the frozen pretrained representation over budget-matched penalised Cox regression, median difference +0.0283. The authors mark the boundary themselves: given the entire training set, Cox regression was favoured in seven of the nine rare cancers, and on its own pretraining objective the encoder lost to a field-frequency baseline by +1.4848 in negative log-likelihood, so reconstruction quality did not predict transfer.

#Applications

SEERFound targets prognostic modelling where labelled outcomes are scarce — rare solid tumours, rare haematological malignancies and paediatric cancers. Because the encoder is frozen and the downstream model is a linear Cox head over 23 routinely coded fields, a group with a few hundred labelled patients can fit a survival model on inputs a registry abstractor already records. All evaluation is internal to SEER 17 on patient-disjoint splits, and the authors state that clinical utility is not established.

#Impact

The interest of SEERFound lies less in the size of its gains than in the shape of the experiment. Comparing against an architecture-identical randomly initialised frozen encoder separates the contribution of pretraining from that of the architecture, a control structured-data foundation model reports often omit, and retraining once per withheld disease tests transfer at the level of the disease rather than the patient. Its negative results inform as much as the positive ones: reconstruction loss ran opposite to transfer, clustering stability missed its reference point, and Cox regression reclaimed the lead once labels were plentiful. Together they locate a regime — a few hundred labelled patients at most — in which outcome-agnostic registry pretraining earns its machinery, and set external validation across registries as the next step.