AGP Microbiome Transformer

Susceptibility to Campylobacter jejuni infection depends on which other species are present in the gut, so the functional role of a bacterium is not a property of the bacterium alone. Yet the standard way to embed a 16S microbiome sample assigns each taxon one fixed vector, learned from global co-occurrence statistics, and represents the sample as an abundance-weighted average of those vectors. Context is averaged away before any downstream model sees it.

This transformer encoder, built at Oregon State University, treats a sample the way an NLP model treats a sentence. The amplicon sequence variants (ASVs) in a sample are ranked in decreasing order of abundance to form an ordered token list, a CLS token is prepended, and self-attention lets every taxon's representation be rewritten in light of every other taxon present. The same ASV therefore receives different embeddings in different communities, and the CLS representation becomes a whole-sample summary that accounts for interactions rather than marginal abundances.

Pretraining happens once, on unlabeled American Gut Project data, and downstream use keeps the encoder frozen — only a projection layer and a fresh classification head are trained. The published version is candid about what this buys: in-domain accuracy is comparable to representation-learning baselines, while the gain appears when the model is pointed at cohorts collected by other studies. Neither the preprint nor the peer-reviewed paper gives the model a name; it is cataloged here under a descriptive slug.

#Key Features

• Contextualized taxon embeddings: Self-attention rewrites each ASV's vector according to the rest of the community, so a taxon is represented differently in different samples.
• Frozen universal encoder: During fine-tuning the embedding layer and all encoder blocks stay frozen; only the projection layer and classification head are trained, which the authors found reduces overfitting and improves generalization.
• ELECTRA pretraining: A generator fills in 15% masked taxa and a discriminator learns to spot the substitutions. The discriminator's encoder is what gets reused downstream.
• Cross-study robustness: An ensemble of ten classification heads, each fine-tuned for a single epoch, replaces held-out-validation early stopping, which is unstable under the distribution shift between cohorts.
• Emergent biological structure: The contextualized embedding space clusters taxa by phylum and correlates more strongly with KEGG metabolic pathways than the input vocabulary embedding, despite no exposure to taxonomy or pathway labels.

#Technical Details

Five encoder blocks operate on a 200-dimensional hidden space, reached by a linear projection from a 100-dimensional vocabulary embedding over 26,726 ASVs, for roughly 7.07M parameters. That vocabulary embedding is preset from previously published GloVe taxa vectors and kept frozen throughout — only the CLS embedding is learned — so the contextualization is what this work trains. Inputs are truncated to the 512 most abundant taxa, which affects under 6% of training points. Pretraining used 18,480 American Gut Project v4 16S samples (16,000 training, 2,480 validation): the generator ran 240 epochs, the discriminator 120, with progressively stronger generator checkpoints swapped in every 15 epochs. The whole run took 23.43 hours on a single Nvidia A40.

On in-domain AGP tasks the transformer reaches AUROC 0.687 for IBD, 0.619 for fruit intake and 0.700 for vegetable intake, ahead of the abundance-weighted GloVe baseline (0.646, 0.585, 0.695) but behind a random forest run directly on raw abundance tables (0.717 for IBD). The picture inverts on independent cohorts: applied without any fine-tuning on their data, the ensemble scores AUC 0.805 on Halfvarson and 0.682 on HMP2, against 0.752 and 0.668 for the weighted baseline and 0.530 and 0.657 for the random forest. Halfvarson shares only 34% of its ASVs with AGP, versus 51% for HMP2.

#Applications

The frozen encoder gives microbiome studies a reusable sample representation for cohorts far too small to train a deep model from scratch — disease status, diet, and other host phenotypes read off 16S ASV tables. Feature ablation attribution on the fine-tuned IBD ensemble ranks which taxa drive a prediction; cross-checked against a curated human gut marker repository, it recovered known markers and flagged Allisonella and Methanosphaera as candidates not previously listed.

#Impact

Most biological transformers encode sequence — nucleotides or amino acids. This one encodes a community, with one token per organism, which is what lets it model interactions among taxa instead of within a molecule. It is small by the standards of the catalog's other microbiome models, such as MGM and Microbiome Self-Supervised Learning, and the authors concede the training corpus is limited in both volume and breadth, being human gut 16S data from a single crowdsourced project. The maintained code lives on Zenodo rather than GitHub: pretrained checkpoints, the fine-tuned ensemble, processed data, and notebooks ship as archives on Dryad and Zenodo. An older, unlicensed copy of the training code sits on GitHub at rsvarma/microbiome_transformers, last updated in 2021 and predating the ensemble experiments.