Loading…
Patient-level single-cell foundation model that condenses a donor's scRNA-seq profile into one 288-dimensional embedding for disease cohort search.
No providers recorded yet. Browse all providers
A single-cell atlas of a diseased heart resolves dozens of cell types, but the clinical question is about the donor: is this patient's transcriptome closer to the dilated cardiomyopathy cohort or the healthy one? Answering it conventionally means comparing two sets of clusters, because scRNA-seq pipelines stop at cell- and cluster-level summaries and never emit an object standing for the person. mcBERT closes that gap by treating a donor's cells as the token sequence of a transformer and pooling the encoder output into one vector per patient.
The model comes from the Communication and Distributed Systems (COMSYS) chair at RWTH Aachen University with two medical clinics at University Hospital RWTH Aachen. Where cell-level models such as scGPT, Geneformer and scBERT tokenize genes within a cell, mcBERT — multi-cell BERT — raises the abstraction one level: each token is a whole cell, and the sequence is the donor. Positional embeddings are omitted outright, since dissociated cells carry no meaningful order.
Two training stages are needed. Self-supervised pretraining teaches the encoder how a donor's cells relate to one another; a supervised contrastive stage then reshapes the space so cosine similarity between patient vectors tracks shared disease. Neither suffices alone — pretraining by itself does not separate phenotypes, and contrastive training from scratch converges poorly.
Each cell enters as a 1,002-wide vector — the 1,000 most highly variable genes plus classification and masking tokens — passes through a linear embedding layer to width 288, and is contextualized by 12 transformer blocks of 12 attention heads each, for 25,572,384 parameters. Pretraining follows data2vec: 15% of a donor's cells are masked, and a student predicts the teacher's normalized average of the top six encoder blocks under a smooth L1 loss, the teacher being an exponential moving average of the student. One tissue takes about 24 hours on a single 96 GB H100, after which SupCon fine-tuning runs with AdamW at a learning rate of 1e-5.
The corpus spans 7,063,540 cells from 1,223 donors across four tissues: heart (six datasets, 199 donors), kidney (five, 80), PBMC (three, 460) and lung (the full Human Lung Cell Atlas, 484). On a single heart dataset mcBERT reaches an ARI of 0.766 and a silhouette score of 0.635, against a raw-counts baseline at 0.018 and 0.034. On the unharmonized union of all heart datasets the gap widens, ARI 0.789 versus 0.055. Averaged across the four tissues, the margin between same-disease and different-disease cosine similarity rises from 0.063 to 0.455.
The natural use is similar-patient retrieval: embed a new donor's cells, then rank a labelled cohort by cosine distance to surface people with comparable disease biology. A k-NN classifier over those anchors assigns a phenotype without further training, and hierarchical clustering recovers disease groups and their sub-structure — the heart embeddings place hypertrophic, dilated and arrhythmogenic cardiomyopathy in neighbouring sub-clusters. Because the vector abstracts away from raw gene expression, the authors propose it as the exchangeable unit in a privacy-conscious patient-comparison service across hospitals.
mcBERT is one of the few attempts to make the individual, rather than the cell, the object a transformer represents in transcriptomics, and it shows that a thousand cells and a thousand genes suffice to place a person in a disease-structured space. Its scope is narrower than the framing suggests: checkpoints are tissue-specific — separate models for heart, kidney, lung and PBMC — and the gene space is the per-tissue intersection of highly variable genes, so a checkpoint transfers only to data sharing that list. A multi-tissue model is named as future work. In the leave-one-dataset-out experiment the held-out heart dataset still contributes unlabelled cells to the self-supervised stage, so what is demonstrated is transfer to unseen disease labels rather than to a cohort absent from training altogether. The code is MIT-licensed, no trained checkpoints have been published, and the work remains a preprint. The name is shared with two unrelated models called MC-BERT — one for Chinese biomedical text mining, one a general-domain NLP method — neither touching transcriptomics.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.