VaxLLM

The VIOLIN vaccine knowledgebase records 4,708 vaccines across 217 pathogens and non-infectious diseases, every one of them read and annotated by hand from roughly 5,000 peer-reviewed articles. Sixty of those entries are Brucella vaccines — and none of the Brucella vaccine papers published from 2023 onward had made it in, because curation is bounded by how fast a human expert can read.

The obstacle is less about reading than about vocabulary. Ask a general-purpose language model for the "vaccine type" described in an abstract and it will answer correctly but inconsistently: "Live-attenuated", "Live bacteria-based vaccine", "Live modified vaccine" — three strings for one controlled term, none ingestible until a curator normalizes it. VaxLLM is instruction-tuned on VIOLIN's own annotations, so the fields it emits come back already phrased the way the database and the Vaccine Ontology phrase them, structured for direct integration after a verification step.

VaxLLM comes from the He Group at the University of Michigan, with collaborators at the Chinese University of Hong Kong, Shenzhen, the Chinese Academy of Medical Sciences and Peking Union Medical College. It is not pretrained from scratch: it is an instruction fine-tune of Meta's Llama-3-8B-Instruct, inheriting its language competence from that base while its domain behavior comes from a small, expert-curated instruction set.

#Key Features

• Two-stage prompt design: One prompt asks whether an abstract carries enough vaccine-formulation detail to count as a Brucella vaccine paper; a second extracts itemized fields from the abstracts that pass, so screening and annotation are separable steps in a pipeline.
• Curator-aligned output fields: Vaccine introduction, type, antigen, formulation, host species used as laboratory animal model, and the experiment used to investigate the vaccine — the same slots VIOLIN records, returned in a fixed response template.
• Standardized terminology: Because the tuning targets were VIOLIN annotations, the model returns the database's canonical terms where the untuned base model returns near-synonyms and appended commentary.
• Fixed-checkpoint inference: New abstracts run through a Hugging Face text-generation pipeline with sampling disabled and no per-corpus refitting.
• Downstream pipeline integration: The annotation run is paired with PubTator entity recognition to cross-check host species and to assemble a host-gene list for DAVID enrichment analysis.

#Technical Details

VaxLLM is an instruction fine-tune of Meta-Llama-3-8B-Instruct, trained with LLaMA-Factory using LoRA with 4-bit quantization and FP16 mixed-precision, a learning rate of 5e-5, per-device batch size 2 and gradient accumulation of 4. The tuning data is small and hand-built in Alpaca format: 150 classification examples (50 positive abstracts corresponding to VIOLIN-annotated Brucella vaccines, plus 100 negatives — 10 unrelated and 90 Brucella-related but without vaccine formulation detail) and 50 annotation examples derived from the same VIOLIN records. Generation is capped at 200 new tokens with sampling off. What ships on Hugging Face is a merged 8B-parameter checkpoint in BF16, not a LoRA adapter, behind an access agreement.

Evaluation used 148 Brucella vaccine abstracts published in 2023–2024 and disjoint from the training set, with manual curation as the gold standard. Classification gave 60 true positives, 84 true negatives, 7 false positives and no false negatives — accuracy 0.95, precision 0.90, recall 1.0, F1 0.95 — against a ROC AUC of 0.71, versus 0.63 for the untuned Llama 3 baseline; the seven false positives were reviews and diagnostic-assay papers that mention Brucella antigens. Itemized annotation accuracy was 97.9% overall versus 80.0% for the base model, with the largest gains on vaccine formulation (100% versus 62.1%) and vaccine antigen (100% versus 77.6%).

#Applications

The direct beneficiaries are biocurators and vaccine researchers who need structured records out of a literature stream that outpaces manual review: the pipeline searches PubMed by keyword, screens for papers with real formulation detail, and emits a spreadsheet of itemized vaccine properties ready for verification and loading into VIOLIN. The same output supports secondary analysis — PubTator entities from the 58 retained articles yielded 54 unique host genes, narrowed to 37 standardized symbols for a Gene Ontology enrichment analysis in DAVID.

#Impact

VaxLLM demonstrates that a modest instruction-tuning set drawn from an existing curated database can convert a general open-weight LLM into a usable curation assistant, and that the gain over the base model lies as much in output discipline as in extraction accuracy. Its limits are equally concrete. Tuning and evaluation cover Brucella only, on 200 tuning examples and 148 test abstracts; the authors list extension to the rest of VIOLIN as future work, so performance on other pathogens is unestablished. The AUC of 0.71 shows the classifier is more threshold-sensitive than its F1 suggests. The instruction dataset and fine-tuning configuration are described in the paper but not released, the repository ships only inference and cleanup notebooks, and the work remains a preprint awaiting peer review.