PubMedBERT (BiomedBERT)

Tokenize a biomedical abstract with the original BERT vocabulary and naloxone comes apart into four word pieces, acetyltransferase into seven. Those pieces carry no biomedical meaning, and the model has to spend capacity reassembling them. The vocabulary was learned from Wikipedia and BookCorpus, and every biomedical BERT built by continual pretraining inherits it, because continuing to train on PubMed text adds parameters' worth of domain knowledge without ever rebuilding the subword inventory underneath.

PubMedBERT's argument is that when a domain has enough unlabeled text of its own, the general-domain starting point should be discarded rather than adapted: derive the WordPiece vocabulary from PubMed, then pretrain from scratch on PubMed alone. The ablations bear this out: swapping the in-domain vocabulary for the Wikipedia one costs 1.2 points of benchmark score, and pretraining first on general text and then on PubMed does no better than skipping the general stage entirely.

The model was built by Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao and Hoifung Poon at Microsoft Research, released in August 2020 and published in ACM Transactions on Computing for Healthcare in 2021. The same paper introduced BLURB, the Biomedical Language Understanding and Reasoning Benchmark, to make head-to-head comparison of biomedical pretraining strategies possible at all. Microsoft renamed the model BiomedBERT in November 2023; the checkpoints are unchanged, and both names refer to the same weights.

#Key Features

• In-domain subword vocabulary: The 30,522-token uncased WordPiece vocabulary is generated from PubMed rather than inherited, so common biomedical terms survive tokenization intact and inputs are substantially shorter — 75.9 word pieces per DDI instance against 106.0 with the general-domain vocabulary.
• Pretraining from scratch, not continual pretraining: Training uses purely biomedical text, avoiding the negative transfer and vocabulary lock-in that come with initializing from a general-domain checkpoint.
• Two corpus variants: One checkpoint is pretrained on PubMed abstracts only, a second adds PubMed Central full-text articles, letting downstream users trade abstract-style precision against broader coverage.
• BLURB benchmark and leaderboard: Thirteen public datasets across six tasks — NER, PICO extraction, relation extraction, sentence similarity, document classification and question answering — scored as a macro average so that NER's many datasets do not dominate.
• Permissively licensed checkpoints: All released weights are on HuggingFace under the MIT license, which is why so much downstream biomedical work loads them directly.

#Technical Details

PubMedBERT uses the BERT-BASE configuration: 12 transformer layers, 768-dimensional hidden states, 12 attention heads, roughly 109 million parameters, and a 512-token maximum sequence length. Pretraining ran on 14 million PubMed abstracts — 3.2 billion words, 21 GB, after discarding abstracts shorter than 128 words — using masked language modeling with whole-word masking at a 15% rate. Optimization used Adam with a slanted triangular schedule peaking at 6e-4 for 62,500 steps at batch size 8,192, about five days on a single DGX-2 with 16 V100 GPUs.

On BLURB the model scored 81.16 macro-average, against 80.34 for BioBERT, 78.86 for SciBERT, 77.29 for ClinicalBERT, 76.27 for BlueBERT and 76.11 for uncased BERT-base. Adding PMC full text (16.8 billion words, 107 GB) scored 81.01 at equal compute and 81.50 when pretraining ran 60% longer, so the full-text checkpoint is a modest gain rather than a clear win. Adversarial pretraining slightly reduced the score, to 80.77. A follow-up study scaled the recipe to the BERT-LARGE configuration and released that checkpoint under the same name.

#Applications

The model is the encoder underneath biomedical text-mining pipelines: tagging gene, chemical and disease mentions in abstracts, extracting PICO elements from clinical-trial reports, classifying chemical-protein and drug-drug interactions, scoring sentence similarity, and answering yes/no research questions. Beyond text, it is routinely loaded as a frozen or lightly fine-tuned text tower in multimodal biomedical systems — BiomedCLIP pairs it with a vision transformer over PubMed Central figure-caption pairs, and ProtST aligns it with a protein language model so that free-text function descriptions can supervise sequence representations.

#Impact

PubMedBERT settled a methodological question that had been assumed rather than tested, and domain-specific pretraining from scratch became the default for fields with large in-house corpora. BLURB outlived the model as the standard yardstick for biomedical encoders, and the leaderboard has since moved on: BioLinkBERT-Large holds the top score at 84.30. The checkpoints remain among the most-downloaded biomedical models on HuggingFace. The limitations are those of its generation — it is encoder-only, so it cannot generate text the way BioGPT can; its 512-token window suits abstracts rather than full papers; and because it never saw clinical notes, it is out of domain for de-identified EHR text, which the paper argues deserves a separate benchmark of its own.