DistillPath

A whole-slide image is cut into thousands of tiles, and every one passes through an encoder before any downstream model sees a prediction. When that encoder is a billion-parameter vision transformer, the cost recurs on every slide and every machine, and the embeddings all have to be stored. DistillPath is the compact alternative: a family of 22M-parameter ViT-S/16 tile encoders from ETH Zurich that inherit the representational behavior of a large pathology foundation model without inheriting its size.

What separates the recipe from earlier pathology distillation is what it is permitted to touch. H0-mini distills H-optimus-0 through the teacher's DINO and iBOT heads, and Virchow2G-Mini uses a DINOv2-style recipe over a billion tiles — but released encoders rarely ship their pretraining heads, and a billion-tile corpus is out of reach for most groups. DistillPath reads only the teacher's final class and patch tokens, so it applies to any released backbone and fits on one consumer GPU.

The students start from the pathology-pretrained kaiko ViT-S/16, supervised by four frozen teachers spanning 86M to 1.1B parameters: Virchow2, UNI2-h, H-optimus-0 and H0-mini. Ramon Kaspar, Andrey Ignatov and Valentina Boeva describe the family in a preprint accepted at the MedFM-Bench workshop at ECCV 2026.

#Key Features

• Backbone-token distillation: The student is supervised by a cosine loss on the class token, a relational RKD loss matching pairwise distances and triplet angles across the batch, and a cosine loss on patch tokens after the teacher's grid is resized to the student's. Teacher and projector are discarded after training, so inference runs the student backbone alone.
• Teacher rank does not predict student rank: H-optimus-0 is the largest teacher and scores 0.803 on the EVA mean, yet gives the weakest EVA student at 0.769, while the 86M H0-mini yields the best gene-expression and robustness students.
• Drop-in timm checkpoint: Each released student is a standard vit_small_patch16_224 state dict with 384-dimensional features, loadable from the Hub through timm or from an exported .pth file without the DistillPath package.
• Measured inference economics: At batch size 64 on an RTX 4090, the best variant is 26.6x faster than Virchow2 and 44.2x faster than H-optimus-0, uses 12.2x less peak CUDA memory, and needs 3.3x less fp32 storage per million tile embeddings.

#Technical Details

Training streams 256x256 tiles online from 6,000 public TCGA diagnostic slides across 32 cancer cohorts, sampling at 0.25 to 2.0 microns per pixel with CLAM tissue detection and HED color augmentation before resizing to 224x224. Each run covers 50,000 steps at batch size 256 in bfloat16 with AdamW, about 24 to 29 GPU-hours on one NVIDIA RTX 4090. Repeating the setup with a generic ImageNet-21k student lifts its EVA mean from 0.729 to 0.754–0.768, so the loss needs no pathology-pretrained start.

Every checkpoint is evaluated frozen under three external protocols. On the seven-task EVA mean, all four variants improve the undistilled kaiko baseline of 0.764; DistillPath-KS16-Virchow2 is strongest at 0.795, within 0.015 of the 0.810 Virchow2 reference at roughly 29x fewer parameters, and above H0-mini at 0.784 and GPFM at 0.789. That lead is task-concentrated rather than uniform — it comes chiefly from BreakHis, rising from 0.720 to 0.849, while MHIST drops and the variant trails H0-mini on five of the seven tasks. On HEST gene-expression regression the H0-mini-distilled student is best at 0.387 mean Pearson correlation against the baseline's 0.349, and it scores 0.495 against 0.307 on the PLISM staining-and-scanner benchmark.

#Applications

The natural use is any pathology pipeline where tile encoding is the bottleneck: cohort-scale feature extraction for weakly supervised slide classification, biomarker prediction, retrieval across scanners and staining protocols, and spatial gene-expression regression from histology. A 384-dimensional ViT-S/16 lets a full cohort be embedded on a workstation GPU or a laptop, with a feature store a fraction of the size a 1536-dimensional encoder produces. The recipe is the second application: any lab with a released teacher and a few thousand slides can build its own student.

#Impact

DistillPath shows that compact pathology encoders need neither proprietary data nor a self-supervised run from scratch — a 22M student trained on public slides for a day of single-GPU compute closes most of the EVA gap to encoders 29 times its size. The four-teacher comparison is the more durable contribution, showing that transfer into a fixed student is not predicted by teacher size or teacher score. The paper states its limits plainly: one student architecture, and TCGA-only training data less diverse than the corpora the teachers saw. The kaiko-initialized checkpoints inherit non-commercial license terms, and of the eight checkpoints described, the H-optimus-0 and UNI2-h students are published under both initializations while the Virchow2 and H0-mini students are not.