BioMedGPT-10B

Paste a raw amino acid sequence into a general-purpose chat model and ask what the protein does, and the answer is usually a request for something else. Llama2-7B-Chat, given UniProt entry Q9LW62, replies "To determine the function of the protein, we need to analyze its structure and sequence. Can you provide me with more information about the protein?" The sequence is right there in the prompt; the model has no way to read it, because a string of one-letter residue codes carries almost no signal a text tokenizer can exploit. The same is true of a SMILES string, where the tokenizer sees punctuation rather than a ring system.

BioMedGPT-10B answers this by giving the language model dedicated eyes for each modality. Molecules enter through a graph encoder and proteins through a protein language model, and a single fully-connected adaptor per modality projects every atom or residue embedding into the token feature space of the biomedical language model, where they sit inside the prompt as <molecule> and <protein> spans. A user then asks an ordinary English question about that object. Developed by PharMolix and the Institute for AI Industry Research (AIR) at Tsinghua University and released in August 2023, the work publishes two checkpoints: BioMedGPT-10B, the aligned multimodal model, and BioMedGPT-LM-7B, the biomedical language backbone it is built on.

The name is shared with BiomedGPT, an unrelated vision-language model for biomedical imaging from a different group; the two models have no components, authors, or training data in common.

#Key Features

• Encoder-plus-adaptor alignment: A 5-layer, 1.8M-parameter GIN initialized from GraphMVP encodes 2D molecular graphs and ESM-2-3B encodes protein sequences. Only the encoders and the two adaptors are trained; the language model stays frozen, following mPLUG-Owl, which avoids catastrophic forgetting of the biomedical text knowledge.
• A separately usable language backbone: BioMedGPT-LM-7B is a Llama2-7B-Chat checkpoint continued on biomedical literature, released under Apache 2.0 and usable on its own as a text-only biomedical LM.
• Two released alignment corpora: PubChemQA pairs 325,754 molecules with 365,129 molecule-text pairs; UniProtQA covers 569,516 proteins across 1,891,506 question-answer samples spanning function, official name, protein family, and subcellular location.
• Interchangeable modality encoders: Because alignment happens entirely in the adaptor, the molecule and protein encoders can be swapped for other pretrained models without retraining the language model.

#Technical Details

BioMedGPT-LM-7B was produced by continued autoregressive training of Llama2-7B-Chat on biomedical literature drawn from the S2ORC corpus: 5.5 million papers were selected by PMC and PubMed ID, reduced to 4.2 million full-text articles after deduplication, stripped of author blocks, references and chart data, and chunked to yield over 26 billion tokens. Training ran 5 epochs at a learning rate of 2e-5, batch size 192, context length 2,048, in bf16 with FSDP, gradient checkpointing and flash attention, converging over roughly 44,000 steps. Multimodal fine-tuning then ran on the PubChemQA and UniProtQA training splits.

On the biomedical QA benchmarks the fine-tuned model reaches 51.4% on MedMCQA and 76.1% on PubMedQA in domain, against human-expert marks of 90% and 78.0%, and 50.4% on USMLE out of domain without training on its split — the only model of its size in the comparison above 50% there, and 5.1 points ahead of fine-tuned Llama2-Chat. On molecule QA over ChEBI-20 it scores BLEU-2 0.234 and METEOR 0.308, roughly double ChatGPT's 0.103 and 0.161. On the UniProtQA protein QA test set it reaches BLEU-2 0.571 and METEOR 0.754, against 0.344 and 0.707 for Llama2-7B-Chat given the same aligned protein features.

#Applications

The practical shape is a conversational front end for molecular and protein annotation: describing an uncharacterized molecule's chemical class and metabolic role from its graph, summarizing a protein's function, family, and subcellular location from sequence alone, or answering literature-grounded clinical questions. For a wet-lab group holding a hit compound or an unannotated open reading frame, this replaces a chain of database lookups with a question in English. The authors are explicit that it should not be used to provide services to the general public, and that outputs need professional supervision and experimental confirmation.

#Impact

BioMedGPT-LM-7B has become a reusable component in its own right: MutaPLM pairs it with an ESM-2 protein encoder to describe mutation effects, and ECG-LM couples it to a ResNet-18 electrocardiogram encoder — both loading the fixed checkpoint rather than retraining a backbone. The models ship inside OpenBioMed, PharMolix's biomedical toolkit, alongside later releases from the same group including PharMolixFM and the reasoning-oriented BioMedGPT-R1. The paper's own limitations section is candid about the weakest part of the evaluation: BLEU and ROUGE say little about whether a generated description of a protein is biochemically correct, and no domain-expert review was conducted. Licensing is split — the code and the 7B language model are permissively licensed, while the 10B multimodal checkpoint is non-commercial.