BulkRNABert

Bulk RNA sequencing is the transcriptomic assay clinical oncology actually runs at scale — cheap enough for large trials and routine pathology workflows, unlike single-cell or spatially resolved protocols. The catch is statistical: a bulk profile is a vector of roughly twenty thousand gene expression values measured across only a few thousand patients, and that ratio defeats classical methods. Practitioners have historically reached for PCA or non-negative matrix factorization to compress the profile before modeling anything downstream.

BulkRNABert, developed by InstaDeep with collaborators at CentraleSupélec and Université Paris-Saclay, replaces those linear compressions with a pretrained transformer. Its central move is to treat expression as text. Each gene's transcript-per-million value is log-transformed, max-normalized, and binned into one of 64 discrete levels, turning a patient's profile into a token sequence over a fixed gene vocabulary. Masked language modeling then transfers unchanged: 15% of gene tokens are corrupted and the model reconstructs their expression bins from the surrounding transcriptional context. Because expression is permutation-invariant, positional encodings are dropped in favor of learned gene embeddings initialized with Gene2Vec.

BulkRNABert is the unimodal predecessor of MOJO, InstaDeep's later bimodal model that adds DNA methylation, and both ship from the same repository. Where single-cell encoders such as scGPT and scBERT operate on individual cells, BulkRNABert deliberately works at the patient level, which is where prognosis is decided.

#Key Features

• Expression binning as tokenization: Continuous TPM values become discrete bins, so the standard BERT masking objective applies directly to transcriptomics with no architectural workaround.
• Frozen-embedding probing: Cancer-type classifiers and survival heads train on mean-pooled embeddings from a frozen encoder, keeping downstream models small enough for cohorts of a few hundred patients.
• IA³ parameter-efficient tuning: An optional rescaling method adds under 0.07% of the model's parameters and outperforms both frozen probing and full fine-tuning on classification.
• Cross-cohort survival transfer: Pan-cancer survival models retain their accuracy on TCGA cohorts removed from the training set, and improve on per-cohort models for the smallest cohorts.
• Pretraining beyond tumors: Healthy-tissue profiles from GTEx and ENCODE enlarge the pretraining corpus, and a model trained on non-cancer data alone still yields competitive cancer embeddings.

#Technical Details

The encoder is small by design: 4 transformer blocks, 8 attention heads each, embedding dimension 256, totaling about 6 million parameters — an order of magnitude below the variational autoencoder baselines it is compared against. It covers the 19,042 genes common to all three pretraining corpora, discretizing each into 64 bins, and was trained on 12 billion tokens across TPU v4-8 devices. Pretraining data comprises 20,406 non-cancer samples from GTEx and ENCODE plus 11,274 TCGA samples spanning all 33 cohorts.

On pan-cancer classification across the 33 TCGA cohorts, BulkRNABert pretrained on TCGA with an MLP head and IA³ rescaling reaches a test macro-F1 of 0.918 ± 0.006 and weighted-F1 of 0.942 ± 0.004, against 0.849/0.877 for PCA plus an SVM and 0.930 weighted-F1 for the multi-omics CustOmics baseline. For pan-cancer survival, it reaches a Harrell C-index of 0.765 ± 0.011, ahead of a tabular masked-autoencoder baseline at 0.756 and a CustOmics RNA-seq-only weighted C-index of 0.630. Per-cohort survival C-indexes include 0.844 on GBMLGG and 0.703 on UCEC.

#Applications

The practical output is a compact patient embedding that slots into existing oncology pipelines. Groups working on tumor-of-unknown-origin classification, prognostic stratification, or trial enrichment can compute embeddings for their own bulk profiles and fit a small head, avoiding the sample-size problem that makes end-to-end deep models unstable on clinical cohorts. The cross-cohort transfer result matters most for rare cancers, where a per-cohort survival model has too few events to train on but a pan-cancer model does not. Embeddings also cluster related tumor types together, which is useful for exploratory subtype analysis.

#Impact

BulkRNABert established that BERT-style pretraining works on bulk transcriptomics, not just on sequences or single cells, and it seeded InstaDeep's multi-omics line that continues in MOJO. Its limits are worth stating plainly: pretraining and evaluation are anchored on TCGA, GTEx, and ENCODE, so generalization is demonstrated only for the cross-cohort survival experiment; the model is small enough that the authors flag scaling as unfinished work; and the governing repository license is CC-BY-NC-SA-4.0, restricting use to non-commercial, share-alike settings. Neither the training data nor a dedicated data card is redistributed — TCGA remains controlled-access through the GDC portal.