Slide-level vision transformer for histopathology that reads a whole-slide image end to end as up to a million pixel patches via dilated attention.
No providers recorded yet. Browse all providers
A whole-slide image of an H&E section is a gigapixel object: at 20× magnification one slide can be tens of thousands of pixels on a side. A standard vision transformer cannot attend over that many patches, because self-attention cost grows with the square of the sequence length. Computational pathology has therefore worked in stages — segment the tissue, cut it into 256 × 256 tiles, embed each tile with a frozen encoder, then learn an aggregation layer on top. The tile encoder never sees the slide it belongs to, and the aggregator never sees pixels.
LongViT, a technical report from Microsoft Research with a co-author at the University of Washington released in December 2023, removes the staging. It splits the resized slide directly into a flat sequence of 32 × 32 pixel patches — up to 1,048,576 of them for a 32,768 × 32,768 input — projects each linearly, adds learned position embeddings, and encodes the whole sequence with LongNet, a transformer variant whose dilated attention scales linearly with length. The slide representation is the average of the resulting patch representations, so the entire model, from raw pixels to prediction, is trained as one network. No tissue segmentation step is required.
The work deliberately follows HIPT: it uses the same TCGA pretraining pool, the same self-supervised objective and the same downstream evaluation protocol, so the comparison isolates the effect of replacing a hierarchy of local windows with one long sequence. It is distinct from Microsoft's later Prov-GigaPath, whose LongNet slide encoder aggregates tile embeddings from a separate tile encoder rather than reading pixel patches directly.
The encoder is ViT-Small-sized: 12 layers, hidden size 384, FFN size 1,536, 16 attention heads and 32 × 32 patches. It was pretrained with the DINO self-distillation objective on about 10,000 diagnostic whole-slide images from more than 30 TCGA cancer types, using 100 random crops of 1,024–1,536 pixels per slide at 20×, global crops of 1,024² and local crops of 512². Pretraining ran for 100 epochs at batch size 1,024 on 64 V100 GPUs, with dilated-attention segment lengths {64 … 1,024} and dilation ratios {1, 2, 4, 8, 16}. Fine-tuning used 8 V100 GPUs for 10 epochs at learning rate 5e-5, with segment lengths rescaled to each input size.
On 10-fold cross-validated slide subtyping, LongViT reported AUCs of 0.912 (BRCA), 0.958 (NSCLC) and 0.990 (RCC), against HIPT's 0.874, 0.952 and 0.980 under the same protocol. For 5-fold survival prediction from histology alone it reported c-indices of 0.700 (UCEC), 0.636 (LUAD) and 0.641 (BRCA). A resolution ablation shows subtyping and LUAD survival improving steadily as input size grows from 1,024² to 32,768². UCEC and BRCA survival do not follow that trend — UCEC peaks at 1,024² — which the authors attribute to the task data or to the gap between pretraining and fine-tuning resolution.
LongViT targets slide-level prediction from routine H&E whole-slide images: histological subtyping of breast, lung and kidney cancers and risk stratification for patient survival. For computational pathology groups it offers a pretrained starting point that can be fine-tuned on a new slide-level label without building a tile-extraction and feature-caching pipeline, provided they have multi-GPU capacity for million-patch sequences. The released code covers TCGA pretraining, subtyping and survival fine-tuning.
LongViT demonstrated that a gigapixel image can be treated as a single transformer sequence rather than a bag of independently encoded tiles, carrying LongNet's long-context language modeling machinery over to pathology. The same LongNet backbone later became the slide encoder of Prov-GigaPath. Its evidence remains limited: it is an arXiv technical report without peer review, pretraining used a ViT-Small-sized model on public TCGA slides only, and all evaluation is fine-tuning on six TCGA tasks with no frozen-feature or external-cohort results.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.