HIPT

A diagnostic slide scanned at 20× magnification can run to 150,000 × 150,000 pixels. The standard way to feed one to a neural network is to cut it into 256 × 256 patches, encode each patch on its own, and pool the embeddings into a slide-level vector with multiple-instance learning. That pipeline throws away what a pathologist reads at intermediate scale: whether tumor cells are invading surrounding stroma, how lymphocytes sit relative to the tumor bed, the macro-architecture of nested glands. A pooling operator cannot express a relationship between two patches a millimetre apart.

HIPT — the Hierarchical Image Pyramid Transformer, developed by the Mahmood Lab with collaborators at the University of Toronto and the Gates Foundation, and presented at CVPR 2022 — instead treats a whole-slide image the way a hierarchical attention network treats a long document. Words aggregate into sentences and sentences into documents; here, 16 × 16 cell tokens aggregate into 256 × 256 patch representations, patches into 4096 × 4096 region representations, and regions into a slide representation. Because self-attention runs only inside each window rather than across the whole slide, the sequence length at every stage stays near 256 and the model remains trainable on ordinary workstation GPUs.

The key observation is that encoding a 4096 × 4096 region built from 256 × 256 patch tokens is computationally the same subproblem as encoding a 256 × 256 image built from 16 × 16 pixel tokens. Self-supervised pretraining recipes designed for small natural images therefore transfer, unchanged, to the higher-resolution stage.

#Key Features

• Three-stage nested architecture: A patch-level ViT over 256 × 256 images (384-dim embeddings) feeds its CLS tokens to a region-level ViT over 4096 × 4096 regions (192-dim), whose CLS tokens feed a lightweight slide-level aggregator. The stages are one model, not three separate encoders.
• Two levels of self-supervised pretraining: DINO student–teacher distillation is applied twice — once to raw 256 × 256 patches, and again to a 16 × 16 × 384 grid of pre-extracted patch features standing in for a 4096 × 4096 region, with crops taken on that feature grid.
• Label-free slide representations: Averaging the pretrained region-level embeddings of a slide and running k-nearest neighbours — with no supervised training at all — reaches 0.974 AUC on renal cell carcinoma subtyping, matching a fully weakly-supervised CLAM-SB baseline.
• Hierarchical interpretability: Attention heads at the two scales factorize into a joint map. Patch-level heads separate stroma, cells and white space, while region-level heads attend to tumor–stroma interfaces and dense tumor cellularity — phenotypes confirmed by a board-certified pathologist.

#Technical Details

Pretraining used 10,678 FFPE H&E diagnostic slides spanning 33 cancer types in The Cancer Genome Atlas, yielding 408,218 non-overlapping 4096 × 4096 regions at 20× and 104 million 256 × 256 patches. The patch-level ViT-S/16 was trained for 400,000 iterations with AdamW at batch size 256 and a base learning rate of 5e-4; the region-level ViT ran for 200,000 iterations over pre-extracted CLS tokens. The region-level and slide-level transformers are 192-dimensional with four and two blocks respectively.

On 10-fold cross-validated slide-level classification in TCGA, HIPT reaches 0.874 AUC on invasive breast carcinoma subtyping, 0.952 on non-small cell lung cancer and 0.980 on renal cell carcinoma, ahead of CLAM-SB, DS-MIL, GCN-MIL and DeepAttnMISL under identical patch features. The margin widens when labels are scarce: at 25% of training data the gains over the best baseline grow to 3.14%, 8.33% and 1.78%. On survival prediction it reports the best concordance index on invasive ductal carcinoma (0.634), colorectal (0.608), clear cell renal (0.642) and stomach adenocarcinoma (0.570).

#Applications

HIPT is used as a frozen feature extractor: run the two pretrained stages over a slide's 4096 × 4096 regions, then train a small aggregation head on slide-level labels such as subtype, grade, stage or survival. That suits computational pathology cohorts of 100 to 10,000 slides, where end-to-end training overfits. The released checkpoints and the HIPT_4K inference API are packaged for exactly this use, and the accompanying notebook produces the hierarchical attention overlays, which help check that a prognostic model attends to tumor rather than scanning artefacts.

#Impact

HIPT was among the first demonstrations that self-supervised pretraining could produce a useful representation of a whole slide rather than of a tile, and the slide-level framing it introduced recurs in later pathology models — the lab's own UNI, CONCH and TITAN, and Prov-GigaPath from another group — at much larger scale. The authors state its limitations plainly: pretraining and evaluation are both confined to TCGA, which over-represents patients of European ancestry and supplies no independent test cohort; slides with insufficient tissue at 4096 × 4096 were excluded from each cohort; and the final aggregation layer could not be pretrained for lack of slide-level data points. Code and both pretrained checkpoints are public in the repository under Apache 2.0 with the Commons Clause, which permits research use but forbids selling the software or a service derived from it.