scSet

Single-cell foundation models such as scGPT, Geneformer, and scFoundation learn representations of individual cells. Clinical questions, however, are asked about patients: does this sample come from someone with COVID-19, with lupus, with pulmonary fibrosis? Bridging the two requires collapsing a variably sized, unordered set of thousands of transcriptomes into a single vector, and the usual answer — average all the cells into a pseudobulk profile — throws away exactly the compositional and within-cell-type variation that motivated single-cell sequencing in the first place.

scSet, from the Clinical Machine Learning group at MIT with collaborators at the Broad Institute and EPFL, is a diffusion-based autoencoder built for this set-to-vector problem. A transformer encoder with a learnable [CLS] token ingests the cells of one sample and produces a patient embedding; a conditional denoising diffusion decoder is then trained to reconstruct individual cell profiles given that embedding. Because the reconstruction target is the sample's own cells, the whole objective is self-supervised and can be run on the large volume of scRNA-seq data that carries no clinical label at all.

The work was presented at the Workshop on Generative AI for Biology at ICML 2025 and posted to bioRxiv in August 2025. Its central claim is practical: patient embeddings learned this way transfer to clinical prediction and hold up when labeled cohorts are small.

#Key Features

• Set-structured encoding: Self-attention over cells handles variable sample sizes and imposes no ordering, letting the encoder model interactions between cells rather than treating each independently.
• Diffusion decoder as the training signal: Conditioning a denoising diffusion probabilistic model on the patient embedding forces that embedding to retain enough information to regenerate the sample's cell population, not just its mean.
• Label-free pretraining: Training requires only unlabeled samples, which is what makes the approach viable given how expensive clinically annotated single-cell cohorts are to assemble.
• Two transfer modes: The frozen embedding can feed a linear probe or MLP, or the encoder can be fine-tuned end to end jointly with a prediction head.
• Generative introspection: Sampling from the decoder produces synthetic cells for a given patient embedding, which the authors use to check that reconstructed cell type proportions track the true sample.

#Technical Details

Cells enter the model as scVI latent embeddings rather than raw counts, which normalizes batch effects and cuts input dimensionality; the authors retain the 14 scVI latents whose standard deviation exceeds 0.4 across the pretraining corpus. A linear embedding layer projects each cell, a [CLS] token is appended, and stacked transformer encoder blocks with multi-head self-attention, feedforward networks, and layer normalization produce the patient representation. The decoder is a multilayer perceptron noise-prediction network that receives the patient embedding and sinusoidal timestep embeddings through adaptive layer normalization, under a cosine noise schedule. Pretraining used 7,342 samples from the CELLxGENE Discover Census spanning many tissue and disease contexts, with 90% of patients for training and 10% held out.

Evaluation covered three clinical tasks: the HLCA triple task (discriminating normal, COVID-19, and pulmonary fibrosis samples in the integrated Human Lung Cell Atlas), a systemic lupus erythematosus PBMC cohort of 162 patients and 99 controls, and a COVID-19 cohort of 102 patients and 24 healthy donors, all under 10-fold cross-validation scored by weighted F1. scSet reached 0.95 on COVID-19, 0.78 on the HLCA triple task, and 0.95 on SLE, ahead of pseudobulk averaging, cell type fractions and means, k-means summaries, and ABMIL-based set encoders. Ablations removing the diffusion objective or substituting a flow-based decoder both degraded performance, and the advantage widened as the number of labeled training samples shrank.

#Applications

The intended use is patient-level phenotype prediction from single-cell cohorts: disease classification, severity stratification, and subtyping where the discriminating signal lies in cell type composition or in subtle within-population shifts. On semi-synthetic data built from multiple myeloma immune cells, scSet separated patient subtypes defined by equal and opposite phenotypic shifts in helper and cytotoxic T cells — a perturbation invisible to averaged representations. Groups with a modest clinically annotated cohort and access to a large unlabeled corpus are the natural beneficiaries.

#Impact

scSet is an argument that the right unit of representation for translational single-cell work is the sample, not the cell, and that a generative reconstruction objective is a strong way to learn it. The release is an early research artifact rather than a package: the repository is BSD-2-Clause licensed but ships scripts for one clinical task, its README states that they will not run without modification, and no pretrained checkpoint is distributed, so reproducing the patient embeddings requires rerunning pretraining. The preprint has not been peer reviewed. Its dependence on precomputed scVI embeddings also fixes the input space, limiting how much gene-level detail the encoder can access.