CancerGPT

DrugComb holds 718,002 drug-pair synergy measurements across 288 cancer cell lines, and 39 of them come from pancreatic tissue. A tabular synergy model needs molecular descriptors for both drugs, an expression profile for the cell line, and enough rows to fit the mapping between them; in a rare tissue none of the three is there. Training on the well-screened tissues instead fails exactly where the rare tissue's pharmacology is distinctive — which is when the prediction was worth making.

CancerGPT takes the opposite route: discard the features entirely. A screening row — two drug names, a cell line, single-agent sensitivities — is written out as a sentence, fed to a pretrained language model, and the last token's 768-dimensional hidden state is passed to a linear head that answers "positive" or "not positive." The knowledge that makes the guess possible is not in the training rows; it is already in the weights, absorbed from the text corpus the backbone was pretrained on.

The model was built at UTHealth Houston with UT Austin and UMass Amherst, posted as a preprint in April 2023 and published in npj Digital Medicine in February 2024. Two unrelated models carry adjacent names: CancerFoundation, a single-cell drug-response model whose codebase happens to name an internal class CancerGPT, and the expression-forecasting model evoCancerGPT.

#Key Features

• Tabular rows as prompts: The task is restated in natural language — "Decide in a single word if the synergy of the drug combination in the cell line is positive or not" — so a general-purpose language model applies without any tabular encoder.
• Warm-up on common tissues: The released GPT-2 checkpoint is fine-tuned on synergy rows from well-screened tissues before any rare tissue is seen; that adapted checkpoint is what the name CancerGPT refers to.
• Transfer to tissues excluded from training: The seven rare tissues are held out of the warm-up entirely, so evaluation measures transfer to new cohorts rather than a held-out fold. Pancreas is scored zero-shot only, its 39 rows leaving nothing to fine-tune on.
• A frozen backbone is nearly as good: Training only the classification head nearly matches updating all 124M parameters, evidence that the last-layer representation already carries most of the useful prior.
• No structural or genomic input: Drugs and cell lines are identified by name, so compounds with no fingerprint and cell lines with no expression profile can still be scored.

#Technical Details

The backbone is the smallest released GPT-2, 124M parameters, loaded with its pretrained weights and tokenizer, with a linear classification head over the final token's 768-dimensional representation. Warm-up training runs on common-tissue rows split 80/20 into train and validation under cross-entropy loss; optional per-tissue fine-tuning follows for k = 2 to 128 shots, four epochs per shot count at learning rate 5e-5 and weight decay 0.01. Labels come from DrugComb's Loewe synergy score, binarized at a threshold of 5, over 4,226 drugs and 288 cell lines. Rare tissues are those with fewer than 4,000 rows: pancreas (39), endometrium (68), liver (213), soft tissue (352), stomach (1,190), urinary tract (2,458), and bone (3,985).

Evaluation uses AUPRC and AUROC on a class-balanced 20% test split within each rare tissue. Among the language-model arms, CancerGPT is ahead of unadapted GPT-2 and SciFive, and matches the much larger fine-tuned GPT-3 arm except in pancreas, where GPT-3 leads — that GPT-3 arm ran on OpenAI's Ada endpoint, not the 175B davinci model the abstract's parameter figure suggests. It leads overall on liver, soft tissue, and urinary tract, whose expression profiles cluster apart from the common tissues; where the rare tissue resembles the common ones — endometrium, stomach, bone — TabTransformer and collaborative filtering trained on that external data win instead.

#Applications

The practical target is combination screening in cancers where high-throughput data is thin: pancreatic, endometrial, hepatic, and soft-tissue tumors, where a team deciding which drug pairs to plate has little or no prior synergy data to fit on. Because the input is just names and sensitivities, a pair can be scored as soon as it can be described in words, which suits shortlisting candidates for wet-lab confirmation rather than replacing it.

#Impact

CancerGPT is an early demonstration that a general-purpose language model's pretrained weights carry usable pharmacological prior knowledge, and that the knowledge survives the move from prose to a serialized data table — an approach BAITSAO later extended to multi-task synergy regression over the same DrugComb screens. Its limits are worth stating plainly: the output surface is a single binary label per query, with no embeddings interface and no free-text generation, since fine-tuning degraded the backbone's generative ability and the paper's reasoning experiments had to be run on GPT-3 instead. Neither code nor weights were released — the availability statement offers method-specific code on request — so the model must be reproduced from the described recipe rather than downloaded.