CLM-access

Cell language models transplanted the recipe behind large language models into single-cell transcriptomics: treat genes as tokens, pretrain on millions of cells, transfer to downstream tasks. scBERT, Geneformer, scGPT, and scFoundation all follow it. Single-cell ATAC-seq resisted the same treatment. Chromatin accessibility is measured over hundreds of thousands to millions of candidate cis-regulatory elements rather than roughly 20,000 genes; the signal is near-binary and extremely sparse; and there is no equivalent of a gene symbol, so peak sets from different experiments do not line up. The standard workaround — discarding more than 80% of cCREs before modeling — throws away most of the regulatory information the assay was run to capture.

CLM-access, from the Hangzhou Institute of Medicine, Chinese Academy of Sciences, is a transformer foundation model built specifically around those constraints. Rather than treating each region as a token, it sorts all cCREs by genomic position, partitions them into a fixed number of contiguous patches, and treats each patch as one token — so the full accessibility profile enters the model without pre-filtering. Pretraining is masked peak reconstruction: the model predicts, at peak-level resolution inside each patch, whether a region is accessible.

Posted to bioRxiv in August 2025, it is the unimodal scATAC-seq predecessor to the same group's later multimodal model CLM-X, which extends the approach to joint RNA and ATAC inputs.

#Key Features

• Patch-based tokenization: Sorting cCREs by genomic coordinate and grouping them into 2,000 fixed-size patches compresses a million-dimensional input into a tractable token sequence while preserving positional adjacency along the genome.
• Peak-level reconstruction inside patches: Binarized peak values are supplied and reconstructed individually rather than summed per patch, so fine-grained regional information survives the compression.
• Unified region reference: A data processing pipeline maps 2.8 million cells from heterogeneous sources onto one reference of over 1.15 million chromatin regions, addressing the absence of standardized peak annotations.
• Zero-shot and fine-tuned modes: Embeddings are usable directly for batch-corrected clustering, and fine-tuning with batch or label information further improves each downstream task.
• Cross-modal prediction head: A fine-tuning path maps chromatin accessibility to transcript abundance, letting the model be evaluated on how much regulatory signal its representations actually encode.

#Technical Details

The architecture has three parts: an embedding module that converts the patch sequence and the binarized peak matrix into embedding sequences and sums them, a transformer encoder producing cell embeddings and contextual region embeddings, and a peak decoder that reconstructs accessibility. A [CLS] token carries the cell representation. Training optimizes a binary cross-entropy loss over masked peaks; the total model is about 20 million parameters, and pretraining ran for 15 epochs. Ablations reported in the paper drove three design choices: binarizing peak signals markedly reduced learning difficulty relative to raw intensities, per-peak BCE inside each patch outperformed summing signals to a patch-level target, and scaling the corpus from the 0.33M-cell scCLIP dataset and the 1.3M-cell CATlas dataset to the full 2.8M-cell collection improved zero-shot clustering roughly in proportion to data size. Parameter scaling behaved differently: a medium-sized transformer outperformed a larger one, which the authors attribute to overfitting given the corpus size, so the medium configuration was kept.

Downstream results span four tasks. For batch correction across four PBMC scATAC-seq datasets, CLM-access exceeds PCA and Harmony in the zero-shot setting and improves further after fine-tuning with batch information. On cell type annotation over two roughly 70,000-cell datasets (GSE219281 and GSE181346), it reaches accuracies of 0.7635 and 0.7470 with macro-F1 of 0.5437 and 0.6742, ahead of scATAnno and Cellcano on both. For RNA expression prediction from paired data, predicted profiles correlate with measured RNA-seq at Pearson 0.9175 with RMSE 1.4185, ahead of BABEL and MultiVI. Multi-modal integration is evaluated as a fourth task.

#Applications

The model targets groups working with large or multi-batch scATAC-seq datasets, where conventional tools either cannot scale or require aggressive peak filtering first: integrating chromatin accessibility atlases collected across labs and platforms, annotating cell types in epigenomic experiments without a matched transcriptomic reference, and imputing expression for samples where only accessibility was assayed. Because embeddings are batch-corrected without needing cell type labels, the zero-shot path is usable as a preprocessing step for exploratory analysis of new datasets.

#Impact

CLM-access establishes that the foundation model recipe transfers to single-cell epigenomics if tokenization is redesigned around the data's actual structure, and it does so at a modest 20 million parameters rather than by scaling. The evidence base is narrower than the transcriptomic models it is modeled on — cell type annotation is benchmarked on two datasets, batch correction on four, and the preprint has not been peer reviewed. Openness is partial: the training, fine-tuning, and inference code is released under the MIT license, but no pretrained checkpoint is distributed, so using the model means repeating pretraining on 2.8 million cells. The preprint itself is under a CC BY-NC license.