Single-cell foundation model pretrained jointly on about 50 million human and mouse scRNA-seq profiles for cross-species cell-type annotation.
No providers recorded yet. Browse all providers
Most of what is known about a human disease mechanism is first worked out in mice, yet the single-cell foundation models built for each species cannot read the other's data directly. The human Geneformer tokenizes human genes, and Mouse-Geneformer tokenizes mouse genes; comparing a mouse model of kidney disease with human patient cells therefore means running two separately trained networks whose embedding spaces were never aligned, or converting one species' genes to its orthologs and discarding the genes that have none.
Mix-Geneformer takes the direct route: one transformer encoder pretrained on human and mouse cells together, so that both species' transcriptomes land in a shared representation. Yuki Nishio, Takayoshi Yamashita, Keita Ito, Tsubasa Hirakawa and Hironobu Fujiyoshi of the Machine Perception and Robotics Group at Chubu University — the group behind Mouse-Geneformer — posted it as an arXiv preprint in July 2025 and presented it at IIBMP 2025, the annual meeting of the Japanese Society for Bioinformatics, that September.
The second ingredient is a contrastive term. Masked-gene prediction alone teaches the model which genes co-occur within a cell; Mix-Geneformer adds a SimCSE loss that pushes two dropout-perturbed views of the same cell toward the same embedding and away from other cells in the batch, aiming for cell-level representations that stay consistent across the species boundary.
The architecture is a BERT-style encoder with six layers, four attention heads, 256-dimensional embeddings, SiLU activations and a 2,048-token input — the same shape as Mouse-Geneformer. Pretraining used a peak learning rate of 1e-4 with a cosine schedule, a per-GPU batch of 8 and ten epochs. Downstream, each organ dataset was split 80:20 and the model fine-tuned for ten epochs at a peak learning rate of 5e-5 with AdamW; experiments ran on eight NVIDIA A100 GPUs.
On nine mouse organs, the contrastive variant fine-tuned to accuracy equal to or slightly above Mouse-Geneformer on most tissues — 95.6% versus 94.9% on kidney (18 cell types) and 97.5% versus 97.0% on thymus — while the graph-network scDeepSort and autoencoder scVAE baselines scored between 48% and 91%. On nine human organs, the human Geneformer stayed marginally ahead after fine-tuning (for example 91.1% versus 88.0% on liver). Zero-shot transfer was uneven: on human immune, kidney and lung data every model, Mix-Geneformer included, fell to roughly 37–50%, while on placenta the contrastive variant reached 80.7% against Geneformer's 75.2%. The contrastive loss did not help consistently, and the variant without it gave the best zero-shot accuracy on mouse tissues.
The model targets translational single-cell work in which mouse and human data need to be analyzed with one tool: annotating cell types in either species after light fine-tuning, and ranking genes by in silico perturbation, in which a gene's rank is raised or lowered and the cosine shift of the cell embedding toward a disease state is measured. In the paper, simulated overexpression of Spp1, Sgk1, Cfb and Rock2 moved healthy mouse kidney cells toward diabetic kidney disease, deleting Umod moved them toward autosomal dominant polycystic kidney disease, and overexpressing S100B, HSPB1 and HSP90AA1 moved human cells toward an Alzheimer's disease state — each a gene previously implicated in vivo.
Mix-Geneformer extends the Geneformer lineage from one species per model to two species per model, and it tests a concrete hypothesis: that a contrastive cell-level objective helps a shared encoder generalize across organisms. The results so far are modest. Fine-tuned accuracy tracks the species-specific models closely rather than surpassing them, the contrastive term's benefit is inconsistent, and zero-shot transfer to human tissues remains weak, which the authors list alongside computational cost as limitations. The work is a single-version preprint without peer review. The first author's GitHub repository provides the pretraining script, SimCSE utilities, and cell-classification and perturbation notebooks built on a copy of the Geneformer codebase, but it carries no license and ships neither pretrained weights nor the joint token dictionary, so reproducing the model requires retraining it from the two public corpora.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.