GeneLLM (cfRNA)

Cell-free RNA in plasma reflects gene expression across tissues, an attractive substrate for non-invasive cancer screening. The conventional pipeline aligns reads to a reference genome, counts them per annotated gene, and trains a classifier on the resulting matrix — a chain gated at every step by annotation, so reads from repetitive regions, transcribed ultra-conserved regions and other unannotated "dark matter" are discarded before the classifier ever sees them.

GeneLLM removes that dependency by treating the raw reads themselves as the language. Developed by a consortium led by Peking University First Hospital with the Institute of Artificial Intelligence at Beihang University and OxTium Technology, it is a transformer pretrained on unlabelled cfRNA reads tokenised into 7-mers. It learns a vocabulary of recurring sequence patterns it calls pseudo-biomarkers, then describes a patient by how their reads distribute across that vocabulary rather than across a gene list.

Posted to bioRxiv in July 2024 as The Language of Cancer, the work was published in Nature Communications under its current title in July 2026, licensed CC BY-NC-ND 4.0. Its Code availability statement deposits the source code, inference code and trained weights in one Zenodo archive, whose license is recorded inconsistently: the paper names the MIT License, the Zenodo metadata says CC BY 4.0, and the archive contains no license file. Raw sequencing data is in the NCBI Sequence Read Archive.

#Key Features

• Annotation-free classification: Trimmed reads go directly into the model, so no alignment, gene counting or differential expression step stands between the assay and the prediction.
• Pseudo-biomarker vocabulary: Read embeddings are aligned to Gaussian-parameterised prototypes; the fraction of a patient's reads matching each of 1,000 prototypes becomes the classification feature.
• Discovery in unannotated regions: Prototypes are learned from sequence rather than a gene model, so many of the most informative fall outside annotated genes, and the top-scoring ones are cancer-type specific.
• Robust to shallow sequencing: Re-sequencing the same libraries at 1 GB rather than 6 GB preserved accuracy, and in-silico subsampling held every cancer type above 0.96 AUC at 20% of full depth.
• Released as an inference stack: The Zenodo deposit carries the pretrained encoder, prototype centroids, trained classifier ensemble, 7-mer vocabulary and a FastAPI service.

#Technical Details

The pretrained core is a Transformer decoder trained autoregressively, each read a sentence and each 7-mer a token, with triangular masking so a token attends only to its predecessors. The paper describes six stacked blocks; the checkpoint on Zenodo is a twelve-block, 768-dimensional decoder over an 8,196-token vocabulary with a 1,024-token context — 92 million parameters in single precision. Pretraining consumed roughly 20 trillion cfRNA reads over about 15 days on 64 NVIDIA A100 GPUs, every parameter randomly initialised. The weights are then frozen, a patient's roughly 40 million reads are encoded, and those vectors are aligned to trainable prototypes — tuned across 200 to 1,200 candidates and set at 1,000 — whose proportion vector feeds a stacked skip-connection network that outputs the disease call.

The clinical cohort comprised 496 plasma samples: 102 colorectal and 102 stomach cancers from Peking University First Hospital, 81 liver cancers from Sir Run Run Shaw Hospital, 47 lung cancers from Beijing Hospital, and 164 controls drawn from all three centres to calibrate batch effects, split 5:1:4. ROC-AUC on the held-out test set ran from 0.9250 to 0.9962 across cancer types and pan-cancer detection, with 83.0% average accuracy in assigning the correct type. Trained de novo on two public cfRNA datasets using their original splits, the model reached a median pan-cancer AUC of 0.96 against the 0.91 reported by Chen et al., and 0.99 against 0.95 on Yu et al.'s pancreatic ductal adenocarcinoma cohort.

#Applications

The target application is early multi-cancer detection from a single blood draw, at a per-sample cost low enough for screening rather than confirmation; the released classifier returns probabilities over healthy plus colorectal, stomach, lung, liver and pancreatic cancer. The pseudo-biomarkers are outputs in their own right: sequences that discriminate cancers yet map to unannotated regions are candidates for follow-up as transcripts or therapeutic targets, and the authors describe the same framework extending to bulk and single-cell RNA-seq.

#Impact

GeneLLM argues that annotation is a lossy bottleneck in liquid biopsy, and that a sequence model reading raw reads recovers signal gene-level quantification throws away — a claim supported by improved AUCs on two public datasets reanalysed with the same pipeline. Its evidence base is bounded: 496 samples, four cancer types, one group, no independent reproduction. The Zenodo deposit makes the trained model runnable on new samples, but ships preprocessing, encoding and prediction scripts without the pretraining or classifier-training code, so the result can be re-applied rather than rebuilt. The annotation-free formulation is the part most likely to generalise, since nothing in it is specific to cfRNA.