CT Foundation

A chest radiograph is one image; a CT study is a stack of hundreds of axial slices that only means something once they are recomposed into a volume. That difference is why the embedding APIs Google Research had built for 2D medical images — CXR Foundation, Path Foundation, Derm Foundation — did not extend to the modality clinicians reach for in the hardest cases, and why training a 3D model from scratch instead means DICOM series assembly, far larger tensors, and the compute budget that comes with them.

CT Foundation answers that with a hosted encoder that takes a CT volume and returns a single 1,408-number embedding summarising organs, tissues, and the presence of abnormalities. The encoder is frozen: a user never re-trains it, and downstream work is fitting a logistic regression or a small multilayer perceptron on top of the embeddings. Because that head is tiny and the input is a 1,408-vector rather than a volume, the demo classifier trains on a CPU inside a Colab notebook.

The representation is learned from radiology reports. CT Foundation is built on VideoCoCa, a video-text architecture that extends the Contrastive Captioner (CoCa) design by pooling multiple frames into one compact sequence representation. Google trained a medical-image-specific 2D CoCa first, then used it as the basis for VideoCoCa, training on prepared axial CT slices paired with their reports under CoCa's two objectives — a captioning loss against the ground-truth text and a contrastive loss pulling matched image-text pairs together. The reports are a training signal only; at inference the model sees pixels. Version v1.0.0 is dated 2024-10-18, and it is documented in Google's Health AI Developer Foundations technical report.

#Key Features

• Whole-study input: The API sorts the individual DICOM slices, composes them into a volume, and runs inference, so users do not write DICOM preprocessing themselves.
• Report-supervised pretraining: Captioning and contrastive objectives over image-report pairs give the embeddings semantic structure that vision-only pretraining does not supply.
• Anatomy-agnostic coverage: Training volumes span head to extremities, and the embeddings separate eight study types including head/neck, spine, heart, angiography, chest, and abdomen and pelvis.
• Low-compute downstream training: Only the classifier head is fit, so building a task model needs neither GPUs nor a large labeled cohort.
• Hosted research endpoint: Access is a free research API rather than a weight download, and Google keeps no copy of the images it processes.

#Technical Details

Training used a private corpus of 527,078 CT studies with associated radiology reports from 430,772 patients across three major hospital regions in the United States. Google has not published a parameter count; the stated specification is the VideoCoCa base and the 1,408-dimensional output. Evaluation covered seven benchmark tasks, fitting non-linear multilayer perceptrons on frozen embeddings at increasing training-set fractions against a held-constant evaluation set: intracranial hemorrhage, calcifications in the chest and heart, lung cancer prediction, suspicious abdominal lesions, nephrolithiasis, abdominal aortic aneurysm, and eight-way body-part classification. Even with limited training data, all but one of the harder tasks exceeded 0.8 AUC. Label quality varies — lung cancer labels come from NLST with pathology-confirmed outcomes within two years, hemorrhage was labeled by board-certified radiologists, and the rest were extracted automatically from radiology reports.

#Applications

The target user is a research team that wants a CT classifier without a 3D training pipeline. The published demo trains a lung cancer classifier on precomputed NLST embeddings and calls the API on LIDC-IDRI studies, a pattern that transfers to any binary finding with a modest labeled cohort: triage prototypes, incidental-finding detection, retrospective cohort studies, or study-level routing by body part and image quality. One operational constraint shapes adoption — the API reads from a Google Cloud DICOM Store, so evaluating it means moving imaging into GCP under short-lived access tokens the caller grants.

#Impact

CT Foundation carried Google's embedding-API pattern from 2D medical images into volumetric imaging, showing that report-supervised contrastive-captioning pretraining yields CT features general enough to serve neuro, cardiothoracic, and abdominal tasks from one frozen encoder. Its constraints bound how far that reach goes. The license is research use only and explicitly not suitable for product development, patient care, or use as a medical device; no weights are downloadable, so it cannot run inside a private network, and access is granted by form and may be revoked on query volume. The model card notes it has not been extensively validated across scanner manufacturers, and the benchmark figures come from a technical report and a research blog post rather than a peer-reviewed evaluation. CT-FM and FM-CT pretrain on images alone and release weights, and FM-CT names CT Foundation among its comparators. The repository's BibTeX asks users to cite the Med-Gemini paper, which describes a related Google model family rather than this encoder; the technical report says only that CT Foundation was trained with a similar approach.