GP-GPT

What is known about a gene and the disorders it causes is scattered across curated databases that were never written for a language model to read: OMIM holds full-text clinical descriptions of Mendelian disease, dbGaP and DisGeNET hold association evidence, UniProt holds gene products and their molecular functions, and NCBI Gene holds the identifiers that tie them together. A general-purpose model asked which gene underlies a given syndrome answers from whatever it absorbed incidentally during web-scale pretraining, because none of those records exist in a form it was trained on.

GP-GPT closes that gap by converting the records themselves into text. Gene entities, protein entities, molecular functions and phenotype associations are pulled from those five sources and slotted into instruction templates — masked-entity prompts, gene-to- protein function prompts, and gene-protein-phenotype triples carrying both true pairs and constructed negatives — producing a corpus a Llama backbone can be instruction-tuned on directly. The result is a fixed checkpoint that answers genomics questions from prompts rather than a pipeline that must be re-fit for each new dataset.

The model comes from the University of Texas at Arlington with the University of Georgia, Indiana University Indianapolis, and Massachusetts General Hospital and Harvard Medical School, and was posted to arXiv in September 2024. It sits alongside domain-adapted biomedical text models such as BioGPT, which are pretrained on the published literature; GP-GPT is instead grounded in curated database records and evaluated on genomics retrieval rather than general biomedical NLP.

#Key Features

• Curated databases rendered as training text: OMIM, dbGaP, DisGeNET, UniProt and NCBI Gene records are converted into templated natural-language contexts, so structured association evidence becomes something an instruction-tuned model can learn.
• Two-stage instruction fine-tuning: a first stage restores a <mask> token to the Llama vocabulary and trains the model to fill masked gene and phenotype entities; a second stage switches to supervised question-answer pairs.
• Adapter-based training across three backbone sizes: LoRA and QLoRA adapters are inserted into frozen Llama attention layers, updating roughly 7.5% of the parameters and yielding GP-GPT Small, Base and Large.
• Relation determination as text generation: gene-disease association calls are posed as yes/no prompts rather than a classification head, so the same checkpoint serves retrieval and relation tasks.
• Layer-wise embedding analysis: sentence embeddings of gene and disease entities are extracted from hidden layers and projected with UMAP, showing how fine-tuning reorganizes tissue-related entity clusters relative to the untuned backbone.

#Technical Details

The training corpus takes the 192,064 NCBI Gene entries as its entry set and adds 4,401 OMIM gene-phenotype pairs verified against dbGaP, 1,286 UniProt gene-protein pairs, and DisGeNET relation-extraction evidence terms — over 3,000,000 terms total. That becomes 2,457,469 stage-one contexts (1,021 MiB) and 546,926 stage-two contexts (308 MiB), with 10,000 texts held out from each as validation. Fine-tuning used 8-bit QLoRA on Llama 2 7B (GP-GPT Small) and Llama 3.1 8B (GP-GPT Base) and 4-bit QLoRA on Llama 3.1 70B (GP-GPT Large), at a learning rate of 1.4e-05 and batch size 32, with early stopping at 2.75 epochs for the two smaller models and 1.25 for the largest. On the medical-genetics QA evaluation, GP-GPT Large reached BLEU-1 0.141 and gene-to-phenotype recall accuracy 0.533, against 0.019 and 0.392 for GPT-4 and 0.106 and 0.492 for the untuned Llama 3.1 70B; GP-GPT Base posted the highest BLEU at 0.404. On relation determination over a TBGA test set screened against DisGeNET, GP-GPT Small scored the best F1 at 0.473 versus 0.394 for GPT-4, though overall accuracy stayed near 0.31 for every model in the comparison, GPT-4 and the 70B baselines included.

#Applications

The model suits medical geneticists and bioinformaticians who need a conversational front end over gene-disease knowledge: asking which genes are implicated in a phenotype, retrieving a gene's function and protein product in prose, or screening candidate gene-disease sentences from the literature for whether they assert a real association. The released weights load as PEFT adapters on top of the public Llama backbones, so they can be merged and run locally — which matters for groups handling patient genomics data.

#Impact

GP-GPT demonstrates that curated genomics databases can be turned into instruction-tuning material rather than left as lookup tables beside a model, and that doing so measurably improves entity recall over both open Llama checkpoints and GPT-4. The caveat is the relation determination result: the fine-tuned models improve F1 but leave accuracy close to what every baseline achieves, so the task remains largely unsolved. The corpus was assembled by rule-based templating and is not fully manually corrected, the authors note that high-quality descriptive text is missing for most genes and diseases, and adapter weights are published only for the Small and Base variants — the 70B Large model that produced the strongest QA numbers is not among them. The work remains an arXiv preprint.