Precious3GPT

A differential expression table is what an omics experiment actually produces, and it is the one format biomedical language models never see: a model trained on PubMed reads the paper describing an experiment, not the 250 up-regulated genes it returned. Precious3GPT (P3GPT), from Insilico Medicine, is trained the other way around — directly on the tabular output of roughly 1.2 million public omics observations, each carrying its metadata as tags. Its native vocabulary is genes, tissues, diseases, age groups and compounds rather than English.

The representation borrows from Geneformer, which showed that an omics sample can be reduced to a ranked gene list with numerical values discarded, making measurements from incompatible platforms comparable. P3GPT reduces each observation to up- and down-regulated gene lists flanked by tags for tissue, condition, compound, species, dose and age, prefixed by an instruction naming the design: a chemical screen, a case-control study, or a cross-age-group comparison. Fields with no value become explicitly empty tags, which is what makes the model generative at inference. Leave the gene lists empty and it writes a signature for the stated condition; fill them in and empty the drug tag, and it names compounds that would produce them.

Posted as a preprint in July 2024 with collaborators at Brigham and Women's Hospital and Harvard Medical School, P3GPT succeeds Precious1GPT, which paired methylation and expression in one aging clock, widening that lineage into a general generator of omics signatures.

#Key Features

• Prompted experiment simulation: A prompt naming tissue, cell line, condition (an EFO or MONDO term), compound, dose, exposure time, age group, sex and species returns the genes that combination would differentially express.
• Bidirectional generation: The same checkpoint runs metadata-to-signature and signature-to-compound, so a desired transcriptomic change becomes a ranked list of molecules expected to induce it.
• Three omics layers, one model: Expression, DNA methylation and proteomics are values of a data-type tag rather than separate models, and the paper's target identification test shows the three modes agreeing on clinical targets.
• Frozen backbone, added modalities: After omics pretraining the transformer is frozen and three-layer feed-forward modality mappers project knowledge-graph and biomedical-text gene embeddings into its latent space.

#Technical Details

The core is a decoder-only MPT-style transformer using ALiBi positional biases: 36 layers, 36 attention heads, a model dimension of 360, a 600-token context and a 63,740-token vocabulary, totalling 89.4 million parameters in bfloat16. Training covered more than 1.2 million observations and 63,376 biological entities — 25,332 genes, 22,241 compounds, 635 conditions, 300 tissues, 269 cell lines — drawn from GTEx, ARCHS4, LINCS (948,654 perturbation samples), PandaOmics, the 2023 Proteomics Drug Atlas, HERB, CNCB, macaque GEO series, UK Biobank and NHANES-IV, plus 13,439 Gene Ontology and canonical-pathway gene sets, with 18,224 age-annotated human, 23,024 murine and 1,124 macaque samples. Mouse and macaque genes map to their closest human homologs, so one vocabulary serves all three species. Modality mappers were then trained over knowledge-graph embeddings from a heterogeneous graph transformer and GenePT text embeddings.

On a held-out set of 805 LINCS perturbations spanning 582 compounds and 102 cell lines, generated gene lists reached Jaccard similarity of 0.0447 (up) and 0.0387 (down) against the real signatures — roughly three times GPT-4o's and three to four times random. Classifiers trained on its gene embeddings annotated 18 Gene Ontology terms at a mean ROC-AUC of 0.8052, best of five models on 8 terms while trailing on molecular function. Stacking embeddings of the 50 most methylated genes with promoter beta-values gave a blood methylation clock with 4.78 years mean absolute error and R² = 0.90 on 2,824 validation samples, against R² = 0.54 for Horvath's 2013 clock on the same data.

#Applications

P3GPT stands in for experiments that would otherwise need a bench: digital case-control studies, chemical screens, and aging clocks, including a pan-mammalian methylation regressor covering 348 species (R² = 0.68). Its target identification test recovered clinically validated genes for 15 of 24 diseases, and for 7 in all three modalities. In the paper's wet-lab check, signatures separating 20- and 80-year-olds were inverted to propose geroprotectors; of 22 compounds screened in an IMR90 senescence model, five showed significant senomorphic activity, maslinic acid and dapsone among them. Weights load through transformers; the p3gpt library, a HuggingFace endpoint and a Discord bot wrap the prompt format.

#Impact

The lasting result is that 89 million parameters trained on raw experimental output match or beat biomedical text models an order of magnitude larger, such as BioGPT-Large and OpenBioLLM-8B, at reasoning about gene function: for omics tasks the training substrate matters more than scale. Its limits are as specific. Prompts must draw on the published entity list, so anything outside those 60,000-plus tokens cannot be expressed, and the format encodes no ontological distance between entities: colorectal and colon adenocarcinoma are unrelated tokens that yield inconsistent targets. Numeric values are handled poorly. The weights carry a CC BY-NC 4.0 licence barring commercial use, and the work remains a preprint.