CoCoS

CoCoS is a pretraining framework for single-cell transcriptomics that augments masked expression reconstruction with a cell-level contrastive objective. It was introduced in an August 2026 arXiv preprint from a multi-institution academic collaboration whose corresponding author is at the Computer Network Information Center of the Chinese Academy of Sciences, with co-authors at Oxford, Tongji, Hunan, and Cambridge. The trained instantiation the authors report results for is CoCoS-GPC, named for its three components: gene partitioning, contrast-set construction, and competence-gated onset.

The work targets an objective mismatch in single-cell foundation models. Models such as scGPT and Geneformer are pretrained largely by predicting masked expression values, which supervises individual genes but imposes no metric structure on the whole-cell representation. A masked value can often be recovered from a small neighborhood of correlated genes, so low reconstruction error does not guarantee discriminative, transferable cell embeddings — precisely what downstream tasks like cell-type annotation and gene regulatory network inference depend on.

Cell-level contrastive learning is the natural complement, but it does not transfer directly from vision. Perturbing expression values can change the cellular state that defines a positive pair, and because sparse transcriptomes are truncated to different gene subsets, a model can separate in-batch negatives from gene-set composition alone without ever learning gene–value correspondence. Cell-level contrast has appeared before in LangCell, TABULA, and scConcept; CoCoS differs by retaining masked reconstruction throughout and pairing it with view construction, negative design, and onset scheduling built for transcriptomes.

#Key Features

• Co-expression-guided gene partitioning: A weighted co-occurrence graph over highly expressed genes is bipartitioned so that strongly co-occurring genes land in opposite panels, yielding two complementary views of each cell whose measured gene–value pairs are left untouched.
• Identity-matched hard negatives: Negatives are built by permuting expression values while holding gene identities and the masking pattern fixed, so gene-set composition is uninformative and the model must discriminate on gene–value correspondence.
• Competence-gated contrastive onset: The contrastive weight stays at zero until a label-free sentinel cohort shows stable cross-view concordance, low reconstruction error, and preserved embedding diversity, then ramps up on a sinusoidal schedule.
• Frozen-embedding transfer protocol: All downstream results come from frozen whole-cell embeddings scored with k-NN and a linear probe, with no backbone retraining on the evaluation datasets.

#Technical Details

CoCoS-GPC uses the Transformer encoder released with scBenchmark (Qi et al., AAAI 2025) rather than a new architecture; the contribution is the training objective and the resulting checkpoint. Pretraining draws on 1,813,780 rows from the public CELLxGENE collection, of which 1,799,684 are used for optimization after reserving three disjoint held-out cohorts (2,048 rows for readiness assessment, 2,048 for an independent audit, and 10,000 for a partition audit). The reported run trains from scratch for 10 epochs at batch size 128 — 140,600 optimizer updates — with AdamW, a 10,000-update linear warmup to a fixed 2e-4 learning rate, mixed precision, and a single RTX 4090. Each view is capped at 512 genes with a 40% mask ratio, cells need at least 50 observed genes, gene assignments flip between panels with 5% probability per minibatch, and the contrast set includes K = 4 fixed-identity value derangements. The controller activated contrastive learning at step 93,000.

On the ten-dataset cell-forward annotation protocol, frozen CoCoS-GPC embeddings reach 74.82% mean k-NN accuracy and 78.85% mean linear-probe accuracy, against 70.99% and 77.02% for scGPT, 67.10% and 77.39% for the reconstruction-only scBenchmark reference, and 70.38% and 66.22% for a Geneformer V2-104M baseline evaluated under a model-specific top-512 median/rank adaptation. On six BEELINE 1000-gene networks, frozen gene embeddings give the highest mean point estimates among the compared variants (AUROC 0.8733, AUPRC 0.7120). A controlled diagnostic shows the confidence gap between ordinary and identity-matched negatives collapsing from 0.358 to 0.028 when four fixed-identity negatives are added.

#Applications

The intended use is feature extraction: researchers embed scRNA-seq profiles with a frozen encoder and attach lightweight classifiers for cell-type annotation, or pair gene embeddings for regulatory-edge scoring, without fine-tuning the backbone. That suits labs annotating new cohorts against a fixed representation and groups building gene regulatory network hypotheses from expression data. The gene-partitioning and gated-onset recipes are also portable design patterns for adding contrastive supervision to an existing masked-expression pipeline.

#Impact

CoCoS contributes to a growing line of work arguing that reconstruction alone is the wrong objective for learning whole-cell representations, and it does so by supplementing rather than replacing masked modeling. Its caveats are substantial and stated plainly by the authors: results come from a single training seed on modest hardware, the Geneformer row is excluded from ranking because its input encoding is not interchangeable, and per-network GRN winners vary, so the averages indicate competitive transfer rather than statistically significant superiority. The preprint has not been peer reviewed, and no code, weights, or data-availability statement accompanies it, so independent reproduction is not currently possible.