CoM³eT

A single breast-cancer workup yields a contrast-enhanced MRI volume and a stained biopsy slide, and no pretrained encoder reads both. Virchow sees only histopathology; CT-FM sees only CT. Medical vision models are divided a second time by output type: encoders built for sparse predictions such as classification bolt dense adapters on for segmentation and lose accuracy and data efficiency doing it, while the strongest segmentation frameworks are configured per dataset and cannot be pretrained.

CoM³eT (Co-representation Multidimensional Multitask Medical Transformer) closes both divisions by decomposing every medical image into its greatest common denominator — the 2D patch — and then running attention over the set of patches. A Swin Transformer V2 backbone turns each 2D input (a whole chest X-ray, a slice of a CT volume, a tile of a gigapixel slide) into one patch token plus a feature pyramid. A BERT-style image transformer models interactions across that token set while returning exactly one contextualized token per input, which is what patient-level and slice-level classification needs. The pyramid transformer then injects the same global context back into the backbone's multi-scale feature maps through sigmoid attention, so segmentation decoders inherit whole-volume context rather than seeing one slice in isolation. Distances are encoded with a symmetric ALiBi variant counted in patch tokens rather than millimetres, keeping position meaningful across MRI slice spacings and slide magnifications alike.

The model comes from Fraunhofer MEVIS with a German clinical and industrial consortium, and scales the group's supervised multi-task pretraining line — UMedPT and Tissue Concepts — from roughly seventeen tasks to more than sixty.

#Key Features

• One encoder across dimensionalities: 2D images, tomographic volumes and whole-slide images are all groups of patch tokens, so one checkpoint covers X-ray, CT, MRI, ultrasound, endoscopy, microscopy and histology.
• Sparse and dense predictions from shared weights: classification consumes patch tokens and segmentation consumes hyperpixel tokens from the shared decoder, leaving task heads that are a single linear or convolutional layer.
• Partial fine-tuning that fits on one GPU: freezing the vision backbone leaves 2.5% of parameters trainable and lets patch tokens be cached, cutting an update step at 32 patch tokens per image from 2.3 seconds to 1.5 milliseconds. An equivalence test across six tasks found partial and full fine-tuning equivalent (0.839 versus 0.833 mean).
• Federated fine-tuning on hospital hardware: only the small trainable part crosses the network, so three sites on consumer GPUs and ordinary internet links reached a C-index of 0.743 ± 0.012 on prostatectomy recurrence against 0.754 ± 0.012 for pooled-data training.

#Technical Details

Variants differ in backbone size: Tiny (Swin-T, 27.6M backbone parameters), UNICORN and Base (Swin-B, 86.9M backbone plus a 4.2M image transformer and a 2.1M pyramid transformer, ~93M in the shared modules) and Large (Swin-L, ~220M). Pretraining ran in two stages on 10 A100 80GB GPUs in bfloat16: first natural images (ImageNet-21k, ImageNet-1k, and COCO for detection, segmentation and captions), then a medical stage covering more than 100,000 patients — over 10,000 whole-slide images and 100,000 annotated pathology patches, RadImageNet's million-plus labelled radiology images, TotalSegmentator, the Medical Segmentation Decathlon and lesion cohorts in CT and MRI, with captions aligned through the frozen SONAR text model. Each update processes one task at a time and discards its head gradients, holding memory constant as tasks are added. On UNICORN, a frozen-weight benchmark of medical foundation models run on sequestered data, CoM³eT scored 0.442 ± 0.022 overall against 0.357 ± 0.014 for a best-of-field ensemble, 0.482 in pathology (next: TITAN and CONCH at 0.356) and 0.458 in radiology. On breast-lesion MRI it reached 58.62% 3D Dice against nnU-Net's 33.13%, and on ultrasound — a modality absent from pretraining — 75.5% ± 2.6% Dice.

#Applications

The practical draw is a single frozen encoder a clinical research group can adapt without a GPU cluster: tumour segmentation in MRI and ultrasound, aortic vessel segmentation in CT, pneumonia detection on chest radiographs, recurrence prediction from prostatectomy slides and tissue segmentation in microscopy of precision-cut lung slices were all fitted from the same weights. Because partial fine-tuning moves only a few million parameters, the recipe also supports multi-site training on data that cannot leave the hospital.

#Impact

CoM³eT is evidence that supervised multi-task pretraining, pooled across specialties and levels of supervision in one final stage, competes with the self-supervised single-domain recipe that dominates medical vision — it won UNICORN and was the only entry applicable to every vision and vision-language task in it. The caveats are concrete: the work is a preprint; code and weights carry a scientific non-commercial research licence rather than an open one; only the UNICORN variant is published as a downloadable checkpoint, with Tiny, Base, Large and the federated variant described but not distributed; and modalities the architecture would admit, such as video and optical coherence tomography, are absent from pretraining.