CellTok

Single-cell transcriptomics and large language models occupy largely separate modeling traditions: single-cell foundation models learn from expression matrices, while LLMs encode the vast biological knowledge captured in scientific text. CellTok bridges these worlds by turning each cell into a sequence of discrete tokens that a language model can read alongside natural language, enabling a single model to reason jointly over molecular measurements and text.

Developed by Chuxi Xiao, Haiyang Bian, Yixin Chen, Lei Wei, and Xuegong Zhang at Tsinghua University and released as a bioRxiv preprint in October 2025, CellTok tokenizes each cell into discrete codebook tokens using a vector-quantized variational autoencoder (VQ-VAE) and integrates those tokens directly into the LLM's vocabulary through early fusion. Because cell tokens and text tokens share one input stream, the model can interleave biological and textual information within the same sequence rather than bolting a specialized encoder onto a frozen language model.

#Key Features

• Cell tokenization via VQ-VAE: Each cell's transcriptomic profile is compressed into a small set of discrete codebook tokens, giving cells a vocabulary-compatible representation that a language model can process natively.
• Early-fusion multimodal design: Cell tokens are added to the LLM vocabulary so that molecular and textual tokens are consumed in a single sequence, allowing tight cross-modal interaction throughout the network.
• Unified single-cell reasoning: A single model performs tasks spanning cell type annotation through prediction of intercellular communication networks, replacing separate task-specific pipelines.
• LLM knowledge transfer: By building on a pretrained language model backbone, CellTok couples quantitative single-cell analysis with the biological knowledge embedded in large-scale text corpora.

#Technical Details

CellTok combines a VQ-VAE cell tokenizer with a transformer-based large language model built on the Qwen family. The VQ-VAE maps a cell's expression vector to indices in a learned discrete codebook; these indices become new entries in the LLM's token vocabulary, and the language model is trained to process mixed sequences of cell tokens and text tokens. This early-fusion strategy contrasts with adapter- or projection-based approaches that keep modalities in separate encoders, and it lets the model attend across biological and textual context at every layer. The resulting system is evaluated on a range of downstream tasks, from annotating cell identities to inferring cell-cell communication networks.

#Applications

CellTok targets computational biologists who want to analyze single-cell RNA sequencing data through a natural-language interface. Its unified formulation supports automated cell type annotation, exploration of intercellular signaling and communication networks, and other transcriptomic tasks framed as language modeling problems. Because cells and text share one representation, researchers can pose questions about cellular states and receive answers grounded in both the measured expression data and the model's textual knowledge, streamlining workflows that would otherwise require multiple specialized tools.

#Impact

CellTok illustrates a tokenization-first route to multimodal single-cell modeling, showing that discrete cell tokens can be embedded directly into a language model's vocabulary rather than fused through a separate encoder. This early-fusion design positions the model within a growing effort to unify single-cell foundation models with LLMs, and its VQ-VAE tokenizer offers a reusable interface between expression data and language-model architectures. As a bioRxiv preprint awaiting peer review, its benchmarks are reported by the authors, and broader assessment of generalization across tissues, species, and assay protocols will depend on further independent evaluation.