GF-Llama-GC

GF-Llama-GC is a single-cell language model built by the Patrick Tan group at Duke-NUS Medical School with collaborators across Singapore and Yale. It targets a representational problem shared by most transcriptomic foundation models: when a cell is written as gene symbols ordered by expression, adjacent tokens are often unrelated in function — a mitochondrial transcript beside an epithelial marker — so next-token prediction learns from a largely arbitrary ordering. The answer here is not a new architecture but a new encoding on top of an off-the-shelf 8-billion-parameter backbone.

In gene-family (GF) encoding, the expressed genes of each cell are partitioned into 29 manually curated families — 15 covering general cellular programmes, 14 enriched in tumours — with genes ranked by expression within each family and one short sub-sentence emitted per family. Every cell sentence opens with a natural-language metadata clause giving cell type, tissue, and clinical attributes such as Lauren subtype, stage, and H. pylori status. This contrasts with the flat ranked-gene sentences of Cell2Sentence and the rank-based tokenization of Geneformer, scGPT, and scFoundation, which the authors note remain in the 0.05–0.3 billion parameter range.

Pilot models on Llama-3.1-8B and Qwen3-8B backbones established that GF encoding beats ranked-gene encoding on every benchmark tested; the Llama variant was then scaled to a 1.3-million-cell gastric cancer (GC) atlas to yield GF-Llama-GC, posted as a bioRxiv preprint in September 2025 and not yet peer reviewed.

#Key Features

• Gene-family cell sentences: Genes are grouped into functionally defined families before ranking, so neighbouring tokens share biological context. An LLM judge scored GF sentences 28.87% more readable than ranked-gene sentences, and GF training reduced final loss by 0.52 on the Llama backbone.
• Domain-adaptive pretraining, not from scratch: Full-parameter continued pretraining of public 8B backbones on cell sentences, sidestepping the cost of training a billion-parameter single-cell model de novo.
• Zero-shot evaluation protocol: Benchmarks probe the pretrained representation directly — gene-level similarity, co-expression network recovery, gene-program separation, next-token prediction, whole-cell generation — rather than tasks requiring fine-tuning.
• Clinically conditioned cell generation: Prompted with metadata alone, the model generates cell sentences whose marker profiles track tumour versus adjacent normal, disease stage, and Lauren histological subtype.
• In-silico cell removal and transplantation: A sample-level analogue of gene perturbation that deletes or transfers a cell population and measures the resulting embedding shift against a random-cell null.

#Technical Details

The GC atlas assembles 11 cohorts (ten public plus one in-house) into 1,313,051 cells and 36,017 genes across 17 cell types after quality control, doublet removal, and scVI integration. The 29 gene families, curated from Gene Ontology, Reactome, DepMap, OncoKB, and primary literature, span 13,647 unique genes; a 30th "Other genes" family absorbs the remainder, and each cell retains the top 50 genes per family plus the top 500 others. Continued pretraining used Axolotl with a causal language-modelling objective, 4,096-token packed sequences, AdamW at a 2e-5 cosine-scheduled learning rate, bf16 precision, and DeepSpeed ZeRO-3 on 8× NVIDIA H200 GPUs — one epoch over 100,000 sentences for the pilot models, three epochs over 1.3 million for GF-Llama-GC.

GF-Llama was the only model to order gene-set similarities correctly (positive control > random > negative control, Kolmogorov–Smirnov p = 0.01) and the only one to recover coherent cell-type-specific co-expression clusters; Geneformer and the unadapted backbones produced mixed networks, while ranked-gene models grouped genes by shared symbol prefixes, a subword-tokenization artefact. In next-token prediction it placed roughly 40% of probability mass on its top gene and over 80% across its top five, and GF models scored higher BERTScores for whole-cell generation on both backbones. Llama-3.1 outperformed Qwen3 throughout, attributed in part to tokenizer efficiency: Qwen splits a gene symbol into a median of five tokens versus three for Llama.

#Applications

The two demonstrated applications are cell-state discovery and cohort-level attribution. GF-Llama-GC embeddings resolved three functionally distinct T/NK subclusters — memory-like, activated, and cycling — in a GC cohort where an expression-based pipeline showed one overlapping population, and recovered gastric and intestinal epithelial lineages in a held-out intestinal metaplasia dataset. After further continued pretraining, it also separated central-memory T cells in ovarian peritoneal metastases. Across 30 GC patients receiving platinum-based chemotherapy, in-silico removal and transplantation implicated tumour epithelial cells and, less obviously, neutrophils in treatment response: responder neutrophils expressed anti-tumour N1 markers, non-responders the pro-tumour marker TGFB1.

#Impact

GF-Llama-GC is presented as a proof of concept, and its contribution is portable: gene-family encoding is a data-representation strategy independent of backbone, applicable to other diseases and larger models. Its limitations are equally concrete. The 29 families are hand-curated and incomplete, with unassigned genes swept into a catch-all category; evaluation is confined to gastric cancer and two open backbones; and the perturbation results are computational predictions without experimental follow-up. The preprint states that the five checkpoints and the accompanying code will be released on publication alongside the 1.3-million cell-sentence dataset; no weights, dataset, or code are publicly available, and the preprint carries no reuse license.