TissueFormer

Paired histopathology and spatial transcriptomics assays let researchers read tissue morphology and molecular state from the same specimen, but the assays are costly and modeling both modalities at single-cell resolution hits a computational wall: a high-resolution slide can contain over one million cells, and attending over every pair of them scales quadratically. TissueFormer, developed at the Eric and Wendy Schmidt Center of the Broad Institute with collaborators at MIT and Harvard, computes exact all-pair attention across an entire slide in linear time, making cell-resolution whole-slide modeling tractable on standard GPU hardware.

Pretraining aligns image- and expression-derived representations of the same cell across H&E slides paired with spatial transcriptomic profiles. The model then performs "virtual spatial transcriptomics" — inferring cell-level gene expression from a routine H&E image alone — and supplies representations for diagnostic prediction at the cell, region, and slide levels. Histology foundation models such as UNI and Prov-GigaPath attend at the patch level, where each patch aggregates hundreds of cells; TissueFormer attends between individual cells, so its attention maps resolve dependencies that patch-based models average away. It is closer in spirit to cell-centric approaches like MAD, but adds whole-slide global context rather than a fixed local neighborhood view. The work was posted as a bioRxiv preprint in August 2026 and has not yet been peer reviewed.

#Key Features

• Linear-complexity whole-slide attention: Reordering the query-key-value matrix products converts quadratic all-pair attention to linear cost with no approximation, so every cell pair is still accounted for. Inference on a slide averages 0.4 seconds.
• Cell-level rather than patch-level resolution: Each graph node is a 224x224-pixel patch centered on a single cell nucleus at a standardized 0.5 µm/pixel, so embeddings and attention maps address individual cells.
• Scalable contrastive alignment: A contrastive objective aligns image- and expression-based cell representations across all pairs within a slide at linear cost, avoiding the quadratic comparisons standard InfoNCE losses require.
• In-context expression prediction: Expression is predicted by retrieving similar image-expression pairs from latent space, covering new datasets and held-out genes from a few reference slides without retraining.
• Interpretable dependency maps: The all-pair attention matrix scores the learned influence between any two cells regardless of distance, supporting in silico analysis of cell-cell communication from histology alone.

#Technical Details

The histology encoder embeds each cell-centered patch with a frozen pathology encoder (H-optimus-1) into a 1,536-dimensional vector, then passes those embeddings and a spatial adjacency graph into a shallow whole-slide graph transformer combining a self term, degree-normalized local propagation over adjacent cells, and global linear attention. The expression encoder maps a cell's counts onto a trainable gene-embedding table over a unified panel of roughly 20,000 human protein-coding genes, followed by an MLP. Pretraining used HEST-1K: 1,593,892 spot-centered and 15,434,350 cell-centered patches from over 1.2K slides spanning 21 organs, two species, four platforms (Visium, Visium HD, ST, Xenium), and 25 cancer types. A two-stage curriculum ran 1,000 epochs on spot-resolution slides, then 200 epochs on cell-resolution Xenium slides with gene embeddings frozen, on 8 A6000 GPUs over roughly two days.

On held-out Xenium slides, gene-wise mean squared error across the 100 most variable genes averaged 0.396, ranking first for 97 of 100 genes and improving 14.5% over the runner-up H-optimus-1 (0.463). Cross-organ Spearman gains over the best competitor ranged from 4.2% to 46.3% on trained organs and 13.7% to 71.6% across 16 unseen organs; mouse-to-human transfer improved 9.7% to 74.0%. On 36 Xenium lung slides, region-level disease severity reached a Spearman correlation of 0.815 and spatial niche classification a mean ROC-AUC of 0.684, where baselines sat near chance.

#Applications

The most immediate use is inferring spatially resolved expression from archival or routine H&E slides, sidestepping the cost of a spatial assay on every sample. The pretrained representations also support patient stratification, region-of-interest disease-state scoring, identification of disease-associated cell types, pathology feature annotation, and spatial niche classification. Two case studies illustrate the analysis side: in pulmonary fibrosis, attention between activated fibrotic fibroblasts and SPP1+ macrophages tracked disease pseudotime and matched shifts in ligand-receptor co-expression; in breast cancer, attention between macrophages and ductal carcinoma in situ cells stratified tumor cells into subtypes that clustering on expression alone did not separate.

#Impact

The contribution is architectural: exact global attention at single-cell scale was infeasible on whole slides, and making it linear opens cell-resolution modeling to labs without exotic hardware. Caveats apply. The results are self-reported in an unreviewed preprint with no independent benchmarking, and the model depends on a frozen external patch encoder rather than learning image features end to end. Distribution is uneven: the code is on GitHub under the MIT license, but pretrained checkpoints are shared through a Google Drive folder rather than a model hub, and the preprint text carries bioRxiv's all-rights-reserved terms — a restriction on reusing the manuscript, not the software.