RNA&Protein SubCell Embedding

A CD4+ central memory T cell and a type 2 conventional dendritic cell can be transcriptionally similar enough that an embedding built from gene counts alone places them in overlapping neighbourhoods. What distinguishes them is less which transcripts are present than where the resulting proteins sit — mitochondrion, plasma membrane, nucleus — and what shapes those proteins fold into. A transcriptome-only embedding throws both signals away before the model sees them.

RNA&Protein SubCell Embedding is a cross-attention framework from the Institute of Image Processing and Pattern Recognition at Shanghai Jiao Tong University that puts them back. Each gene symbol in the expression matrix is mapped through UniProt to its protein entry, that protein is encoded by FoldExplorer — a graph-attention structure descriptor from the same lab that folds in ESM-2 sequence features — and the resulting vector is concatenated with a seven-dimensional localization prior giving its dominant probability of occupying the cytoplasm, nucleus, plasma membrane, mitochondria, endoplasmic reticulum, Golgi apparatus or endosome. The transcriptomic side is a masked-attention transformer in the mould of scGPT, and the two towers meet in a cross-attention block where RNA embeddings supply the queries and keys that modulate cell-surface protein tokens. A prepended CLS token pooled from that fused block is the cell representation.

The framework is pretrained with its own composite self-supervised objective and the resulting checkpoint is then fine-tuned across four downstream task families, which places it alongside transcriptome-only single-cell foundation models such as Geneformer and scFoundation rather than among per-dataset integration tools. Two components are inherited rather than trained here: the RNA encoder is initialized from scGPT's configuration and vocabulary, and the protein descriptors come from FoldExplorer. The protein-side transformer encoder is initialized randomly.

#Key Features

• RNA-conditioned protein encoding: Cross-attention uses transcriptomic embeddings as queries and keys over protein values, so proteomic representation learning is guided by the expression context of the same cell rather than learned independently and concatenated.
• Structure-aware protein tokens: Protein inputs carry FoldExplorer descriptors encoding folding topology and functional similarity, distinguishing proteins that share an expression level but not a function.
• Compartment priors on the embedding: Annotated locations are mean-pooled into one vector added to each token embedding, so a protein with several compartments gains no magnitude merely from being better annotated, and an unannotated feature receives an exact zero vector.
• A single pooled cell vector: CLS pooling yields one representation encoding intra-protein structure, transcriptomic context and spatial organization, and feeds cell-type annotation, batch correction, multi-omics integration and cell-cell communication heads.

#Technical Details

The released pretraining configuration is a 12-layer, 8-head transformer with an embedding width of 512 and a feed-forward width of 512, reading expression values binned into 51 levels over sequences truncated at 1,200 tokens, with FlashAttention and mixed-precision training distributed across 16 ranks. Genes are masked at ratios of 0.25, 0.5 and 0.75. The pretraining loss is a weighted sum of masked gene expression modelling and cross-modal protein prediction, at 0.6 and 0.4 respectively.

On the 10x Multiome PBMC dataset the fine-tuned model reaches 0.962 accuracy, 0.961 precision, 0.937 recall and 0.947 macro-F1; on a multi-tissue immune benchmark it reaches 0.922, 0.898, 0.920 and 0.905, against 0.790, 0.848, 0.732 and 0.761 for Seurat. An ablation removing the subcellular prior drops 10x Multiome PBMC accuracy from 0.94 to 0.92 and macro-F1 from 0.92 to 0.90. For integration, the model records an AvgBIO of 0.657 on Human PBMCs Covid against 0.626 for scGPT, 0.607 for TotalVI and 0.519 for Scanpy, and on a combined TEA-seq, ECCITE-seq and CITE-seq benchmark an AvgBIO of 0.567 and AvgBATCH of 0.968 against scGPT's 0.517 and 0.951.

#Applications

The framework suits laboratories with jointly measured RNA and cell-surface protein assays — CITE-seq, TEA-seq and ECCITE-seq — that want cell-type labels and batch-corrected embeddings from one representation. The localization module is independent of the released annotation table: any curated source, whether UniProt subcellular-location records, Gene Ontology cellular-component terms or Human Protein Atlas exports, can be normalized into the documented feature-location schema and fused into the token embeddings.

#Impact

The entry is a preprint awaiting peer review. Source code, preprocessing scripts and downstream tutorials are public, but no trained checkpoint has been released, the repository carries no license file, and the preprint does not report the size or composition of the pretraining corpus. Evaluation is confined to immune and PBMC data, so the reported margins do not establish behaviour on other tissues. The authors state two further limits: it models static cell snapshots without trajectories or perturbation response, and its subcellular signal is a coarse annotation rather than a spatial measurement. Its contribution is the demonstration that a seven-dimensional prior, essentially free to compute from existing annotation databases, measurably sharpens immune subtypes that expression alone leaves confused.