GENEREL

Ask PubMedBERT how similar type 1 and type 2 diabetes are and it answers 0.995 — all but identical, because both phrases describe elevated blood glucose. One is autoimmune destruction of insulin-producing cells, the other acquired insulin resistance, and their genetics separate cleanly — but an encoder trained on abstracts cannot see that. Ask the same model about rs2476601 A and it has nothing to say at all: a variant identifier is not a word, and no biomedical language model embeds one.

GENEREL (GENomic Encoding REpresentation with Language model) closes both gaps by training a single vector space that holds free-text biomedical concepts and single-nucleotide polymorphisms side by side. Concepts are encoded by a fine-tuned transformer; each SNP–risk-allele pair gets a row in a learned embedding matrix; and a weighted contrastive objective pulls the two halves into alignment using association evidence — GWAS odds ratios, eQTL effect sizes, biobank phenotype–genotype correlations — as supervision. Because the concept side reads natural language rather than a code, the framework needs no crosswalk between EFO, PheCode, SNOMED CT and ICD: a trait phrase from one source and its synonym from another land in the same neighbourhood. In GENEREL's space, type 1 and type 2 diabetes sit at 0.815 rather than 0.995.

The work was posted in October 2024 by the Department of Biomedical Informatics at Harvard Medical School, with the VA Boston Healthcare System, Brigham and Women's Hospital and Tsinghua University.

#Key Features

• Variants and clinical language in one space: 83,900 SNP–allele concepts and arbitrary biomedical phrases share a 768-dimensional embedding, so a variant can be matched to a disease, drug, pathway or gene by cosine similarity alone.
• Association strength as supervision: the InfoNCE loss is weighted by the odds ratio, regression coefficient or phenotype correlation behind each pair, so the geometry encodes association strength and not merely association.
• Ontology-agnostic on the concept side: any new phrase encodes directly through the language model, removing the manual code mappings graph-based methods depend on and the errors those mappings introduce.
• Four training tasks, four sources: variant–trait links, biobank correlations, knowledge-graph relations and terminology synonyms are optimised jointly, and ablations show each lifts a different benchmark.

#Technical Details

The concept encoder shares SapBERT's architecture and initialises from its weights; a randomly initialised linear layer projects the [CLS] hidden state to 768 dimensions. SNPs are one-hot indices into a separate 768-dimensional embedding matrix, also randomly initialised. Training follows CLIP's InfoNCE implementation with a learnable temperature, over 1,173,911 weighted pairs: 135,749 term–SNP pairs from the GWAS Catalog and GTEx eQTL, 467,026 term–SNP pairs from a UK Biobank EMIS cohort of 216,215 patients spanning 6,358 phenotypes and 61,455 SNPs, 325,324 term–term relations from PrimeKG, and 245,812 synonym pairs from UMLS 2020AB. One run of 25 epochs at batch size 512 on a single L40S GPU produces the checkpoint. The SNP vocabulary is fixed in advance — 65,278 variants and 83,900 SNP–allele concepts, after dropping GWAS Catalog SNPs linked to fewer than two traits and keeping only A/T/C/G risk alleles — so a variant outside that set has no embedding.

Evaluation applies the trained embeddings frozen. On external knowledge bases the model reaches AUCs of 0.760 (DisGeNET disease–gene), 0.758 (pathway–gene), 0.824 (DrugBank indication–drug) and 0.850 (indication–gene), ahead of BioBERT, ClinicalBERT, SapBERT, CODER, KRISSBERT and BGE on all four. On variant–trait retrieval it scores 0.942 and 0.939 on the GWAS Catalog test split against 0.894 for the best graph-embedding baseline. On the Million Veteran Program cohort — an independent source of associations — it reaches 0.793 with traits as anchors, ahead of a correlation-matrix SVD baseline at 0.775, but trails that baseline with SNPs as anchors, 0.786 against 0.840. Substituting trait synonyms costs 0.012 AUC on MVP and 0.065 on GWAS.

#Applications

The practical use is cross-source retrieval: pulling the traits linked to a variant out of one biobank using a phenotype label written for another, prioritising variants for a disease phrased in free text, or ranking drug indications against genetic evidence. Because concepts enter as language, a group holding GWAS statistics coded in EFO can query a cohort coded in PheCode without first building a crosswalk — the step that usually blocks integrative analysis across biobanks.

#Impact

GENEREL is the first framework to place variant-level genetic concepts and clinical language in one learned space rather than bridging them with a curated mapping, and its weighted objective shows effect sizes can be trained into geometry. The limits are worth stating plainly: it remains a preprint, later presented at an ICML 2025 workshop; the SNP side is a lookup table over a curated vocabulary, not a model that generalises to unseen variants; training draws on the controlled-access UK Biobank; and no code or weights have been released, so the embeddings cannot yet be reused directly.