Influ-BERT

Influenza A virus surveillance turns on knowing which subtype a sequenced sample belongs to, and the subtypes that matter most for pandemic preparedness — H5N1, H7N9, H9N2, H5N8 — are precisely the ones with the fewest deposited genomes. Public sequence archives follow a steep long-tail distribution dominated by the seasonal human subtypes H1N1 and H3N2, so classifiers trained on them collapse on the rare zoonotic subtypes. Alignment-based subtyping tools have a different problem: the reference databases they compare against go stale as the virus mutates, and phylogenetic reconstruction scales poorly to millions of sequences.

Influ-BERT, from Rongye Ye, Lun Li, and Shuhui Song at the Beijing Institute of Genomics and the China National Center for Bioinformation, addresses this with domain-adaptive continued pretraining. Rather than train a genomic language model from scratch, the authors take DNABERT-2 — pretrained on 32.5 billion bases across 135 species — build an influenza-specific byte-pair-encoding vocabulary, and continue masked-language-model pretraining on a corpus of roughly 900,000 influenza genome sequences. Task-specific classification heads are then fine-tuned on top.

The model is a classifier, not a generator: it reads sequences and assigns labels for surveillance and diagnostic triage. It was posted to bioRxiv in August 2025 and published in Briefings in Bioinformatics in March 2026.

#Key Features

• Domain-adaptive pretraining: Continued MLM pretraining on an influenza-only corpus, with the BPE vocabulary shrunk from 4,096 to 512 tokens to match the shorter, less diverse viral genomes.
• Low-frequency subtype recovery: The two-stage strategy targets the long-tail problem directly, lifting recognition of subtypes that baseline models miss entirely.
• Segment-agnostic and segment-specific modes: One workflow subtypes from any genomic fragment, useful when sequencing coverage is incomplete; a second is trained only on hemagglutinin and neuraminidase segments for higher-accuracy applications.
• Four downstream tasks from one backbone: The same domain-adapted checkpoint supports subtype identification, respiratory virus discrimination, genomic segment and functional gene identification, and pathogenicity prediction.
• Perturbation-based interpretability: A sliding-window masking analysis quantifies each fragment's contribution to a prediction, showing consistent attention to specific genomic regions across sequences of the same subtype.

#Technical Details

The pretraining corpus comprises approximately 900,000 sequences pulled from NCBI Nucleotide: about 800,000 influenza A, 100,000 influenza B, and over 4,000 influenza C and D, spanning more than 100 naturally observed HA-NA combinations and all segments. Downstream evaluation uses 40,000 independently collected IAV sequences covering 66 subtype combinations, held out from pretraining, split 8:0.5:1.5 into train, validation, and test. All other pretraining hyperparameters follow DNABERT-2; fine-tuning updates all parameters with cross-entropy loss, implemented in PyTorch 2.5.1 and Hugging Face Transformers 4.49.0.

Across the reported tasks Influ-BERT holds F1 above 97% overall. On five-class subtyping it reaches F1 of 97% for H5N1 and 94% for H9N2; on the harder ten-class task it scores 87% on H5N8, a subtype most competing models fail to recognize at all. Baselines are k-mer random forest, SVM, and logistic regression, plus DNABERT-2 and MegaDNA — notably, unadapted DNABERT-2 underperforms classical machine learning on these long-tailed tasks. For pathogenicity the authors deliberately test the data-scarce regime: F1 of 79.5% from 200 labeled samples and 83.3% from 400, ahead of the next-best model by about four points. Two limitations are stated plainly: fine-tuning is closed-set supervised learning, so novel subtypes absent from training cannot be recognized, and training on nucleotide sequence alone means no protein-structural signal is captured. Code and weights, including the pretrained backbone and the fine-tuned task checkpoints, are released under the MIT license.

#Applications

The model suits public health laboratories and genomic surveillance programs that need to subtype large volumes of influenza sequence quickly, without maintaining an up-to-date alignment reference. The segment-agnostic mode is aimed at rapid detection where sequencing depth is limited and HA or NA segments may be missing; the HA/NA mode targets higher-precision work such as vaccine strain selection. The respiratory virus task extends the same backbone to distinguishing influenza from SARS-CoV-2, rhinovirus, and RSV in clinical sequencing panels.

#Impact

Influ-BERT is a clean case study in domain-adaptive pretraining for viral genomics: it shows that a general-purpose DNA language model can actually underperform k-mer baselines on a narrow, long-tailed viral corpus, and that continued pretraining with a domain-matched tokenizer reverses that gap. Its practical value is concentrated where surveillance is hardest — the rare zoonotic subtypes with the greatest pandemic relevance and the least data. Released with permissive licensing and open checkpoints, it is directly reusable, and it complements other influenza-focused sequence models such as AntigenLM from the same institute.