Celler

Assigning a cell type to every cell in a single-cell RNA sequencing experiment is the step that turns an expression matrix into biology, and pretrained models have largely displaced manual marker-gene curation for the job. But the models that do it well are trained on data whose class distribution is badly skewed. In diseased tissue, the cells that matter clinically are the scarcest: in one lung sample from a lung cancer patient, malignant cells account for 0.92% of the population. A cross-entropy objective spends almost all of its gradient on the healthy majority, overfits the head classes, and underfits precisely the tail the experiment was run to find.

Celler, from the School of Computer Science and Electronic Engineering at Hunan University with Higentec AiLab, is a genomic language model built to fix that imbalance rather than to add a new architecture. It follows the masked-gene-expression paradigm established by scBERT, scGPT, CellPLM, and Geneformer — genes as tokens, a cell as a sentence — and keeps that backbone intact. Its two contributions are both training-time: a reweighted loss that enlarges the region of logit space a rare class occupies, and a mining strategy that returns the model to the samples it confuses most.

The work also introduces Celler-75, a pretraining corpus of 41,307,753 cells covering 21,292 genes, 80 human tissues, and 75 diseases including cancers and Alzheimer's disease.

#Key Features

• Gaussian Inflation loss: Perturbs each sample's logits with a draw from a Gaussian whose amplitude scales with how rare its class is, expanding the feature-space footprint of tail categories so head classes stop dominating the decision boundary.
• Hard Data Mining: Identifies the categories that score highest while being wrong for a given cell, then adds a balanced loss term over that hard-category set so training concentrates on genuinely confusable pairs instead of easy cells.
• Hierarchical label granularity: Annotates at both parent-class and subclass levels — immune cell versus T cell — which is where long-tail errors concentrate, since subclass accuracy trails parent-class accuracy across every tissue tested.
• Disease-tissue pretraining: Celler-75 was assembled specifically from diseased human tissue rather than healthy atlases, so the pretrained representation has seen pathological cell states directly.
• Checkpoint-based inference: Annotation runs by pointing a released tissue-specific checkpoint at a new .h5ad file, with no per-dataset retraining step.

#Technical Details

Celler adopts scGPT's gene tokenization, assigning each gene an integer ID and discretizing expression values into bins within a batch to blunt batch effects. Gene identity and expression embeddings feed a Transformer encoder with multi-head attention, pretrained by masking 15% of gene expression values at random and predicting them with an MLP head. For annotation, features from the pretrained encoder pass to a classifier head trained under the combined Gaussian Inflation and hard-mining objective. Parameter counts are not reported.

On the public Multiple Sclerosis benchmark Celler reaches an F1 of 0.799 against 0.703 for scGPT and 0.599 for scBERT; on hPancreas it reaches 0.767 against 0.739 for SingleCellNet, whose precision of 0.761 edges out Celler's 0.755. On a four-organ slice of Celler-75 (800,000 cells, 600,000 for fine-tuning and 200,000 held out), the largest gains are at subclass level — brain subclass F1 of 0.890 against 0.802 for scGPT — while parent-class results fluctuate, with kidney falling to 0.820 behind scGPT's 0.880. A loss ablation on brain data puts the full objective at F1 0.895 against 0.797 for cross-entropy.

#Applications

Celler targets disease-focused single-cell studies where the biology of interest sits in the tail: tumor microenvironment profiling, identifying scarce pathological populations in neurodegenerative tissue, and biomarker discovery that depends on resolving cell subtypes rather than broad lineages. Because inference is a single script against a released checkpoint, a lab with a new .h5ad matrix from one of the covered tissues can annotate it without assembling a training set.

#Impact

Celler is a useful demonstration that long-tail methods from computer vision transfer cleanly to single-cell annotation, and that the gains land where they should — on rare subclasses rather than on aggregate accuracy. Its release is partial: Celler-75 itself is private, with train and test subsets published for seven of the 80 tissues via a third-party cloud link, pretrained weights are distributed per tissue rather than as one general checkpoint, and the repository carries no software license while the dataset terms prohibit commercial use. The work remains a preprint.