FratMAE

PET/CT is the workhorse of oncologic imaging: the CT component supplies anatomy, the PET component supplies metabolic and molecular-marker activity, and clinicians read the two together for staging and treatment monitoring. Deep learning on PET/CT has nonetheless remained a collection of task-specific models trained from scratch on modest cohorts, while large-scale medical imaging foundation models have concentrated on a single modality — chest radiographs, or CT alone, as in Merlin — leaving multimodal PET/CT without a pretrained backbone.

FratMAE, the Cross-Fraternal Twin Masked Autoencoder, was built to fill that gap by the Center for Advanced Medical Computing and Analysis at Massachusetts General Hospital and Harvard Medical School, with nuclear medicine departments in Bern, Cologne and Augsburg. Its design turns the two modalities into each other's supervision: separate Vision Transformer encoders process PET and CT, and cross-attention decoders reconstruct masked tokens of one modality using the intact representation of the other. Anatomy therefore has to explain function, and function anatomy, rather than each being learned in isolation.

Two further choices set it apart from generic 3D masked autoencoders. FratMAE tokenizes coronal rather than axial slice stacks, so a training patch spans the body head-to-foot and the model can learn the global tracer-uptake patterns axial crops truncate. And it treats the clinical context that governs uptake — radiotracer, diagnosis, age, sex — as a text string aligned to the PET representation by contrastive learning.

#Key Features

• Asymmetric cross-modal masking: On each iteration one modality is chosen for masking; when 50% of PET tokens are hidden, the CT volume is left fully visible, and vice versa, so reconstruction must draw on the partner modality.
• Cross-attention reconstruction: In each decoder block the masked modality's representation supplies the query while the unmasked modality supplies keys and values, making cross-modal correspondence the explicit reconstruction path.
• Whole-body coronal patches: Coronal scan stacks replace the axial stacks used by most 3D medical foundation models, preserving head-to-foot context; on segmentation this alone lifts Dice from 0.559 to 0.795 without any pretraining.
• Text-conditioned PET representations: A ContextAlign module encodes a formatted metadata prompt with a CLIP text encoder and aligns it to the PET class token through an InfoNCE objective.
• Data-efficient transfer: The pretrained encoders target downstream fine-tuning under scarce labels, and reported gains are largest with only a fifth of the downstream training set.

#Technical Details

Both encoders and decoders use the standard ViT-B backbone within the masked autoencoder framework, with a ViT-B CLIP text encoder for metadata. Volumes are resized to 160x160x192 and cropped to 3D patches of 32x160x192 voxels with a height subsampling factor of 2. Pretraining combines a mean-squared reconstruction loss with the InfoNCE alignment loss and ran for 30 epochs on eight NVIDIA A100 40GB GPUs. The corpus is the public AutoPET III collection: 1,292 whole-body PET/CT volumes covering melanoma, lymphoma, lung cancer, prostate cancer and negative controls, acquired with 18F-FDG, 18F-PSMA and 68Ga-PSMA tracers.

Downstream evaluation uses 515 18F-FDG volumes from the German Hodgkin Study Group trials, split 433 for training and 82 for testing. Lesion segmentation attaches a UNETR-style convolutional decoder with skip connections; Ann Arbor staging attaches an MLP classifier over early versus advanced stage. Restricted to 20% of the downstream training set, FratMAE reaches a Dice of 0.640 and IoU of 0.496 against 0.597 and 0.454 for a model trained from scratch, and 0.534 and 0.392 for conventional MAE pretraining, with staging accuracy of 0.654 against 0.603. Ablations attribute part of the gain to cross-modal masking and part to ContextAlign. On the full training set the segmentation advantage narrows to a single Dice point.

#Applications

The intended workflow is a shared PET/CT encoder that hospitals fine-tune for individual reading tasks instead of training a separate network per task and per cohort. Lesion segmentation supports tumor burden quantification for treatment planning and response assessment, and automated Ann Arbor staging supports risk stratification in lymphoma. Because gains concentrate in the low-label regime, the approach suits nuclear medicine departments with small annotated trial cohorts. The encoders can also be applied to CT or PET alone.

#Impact

FratMAE is a preprint that has not been peer reviewed, and no code or pretrained weights have been released, so its results cannot be reproduced or its encoders reused by others. Its contribution is nonetheless a concrete recipe for multimodal medical pretraining: making one modality reconstruct the other under asymmetric masking is a more targeted objective than masking both, and the coronal-stack result is a reminder that patch geometry can outweigh pretraining strategy in whole-body imaging. Validation rests on one pretraining corpus and one single-tracer Hodgkin lymphoma cohort, so generalization to other tracers, indications and imaging centers remains to be shown; the authors identify multi-center, multi-tracer expansion and treatment-response prediction as next steps.