Histopathology vision-language model that aligns frozen Virchow2 and BioMedBERT encoders with small LoRA adapters for zero-shot H&E classification.
No providers recorded yet. Browse all providers
A pathology foundation model such as Virchow2 produces excellent tile embeddings, but it cannot read. To use it for a new diagnostic question, someone still has to collect labeled examples and train a classifier. Vision-language models remove that step by placing images and text in one embedding space, so a class can be defined by writing a sentence like "lymph node tissue with metastatic carcinoma". The usual way to build one, however, is to fine-tune large image and text encoders on paired image-caption data. That demands substantial compute and can overwrite the representations the encoders learned in pretraining.
Lumen, developed at the Institute of Tissue Medicine and Pathology of the University of Bern with collaborators at Inselspital, Bern University Hospital and the University of Cologne, tests a lighter recipe. It takes two independently released encoders, the Virchow2 vision transformer from the Virchow family and the biomedical text encoder BioMedBERT (PubMedBERT), and keeps both completely frozen. Only rank-4 low-rank adapters (LoRA) and linear projection heads are trained, about 0.40% of the assembled model's parameters, to pull matched image-caption pairs together in a shared 512-dimensional space. The result is a single checkpoint that answers text-prompted queries on tissue it never saw during alignment.
The model was released as an arXiv preprint in September 2026. Its name collides with an unrelated chest X-ray model, LUMEN; the two share no authors, data or code.
Virchow2 contributes a 2,560-dimensional image embedding (class token concatenated with the mean of the remaining tokens) and BioMedBERT a 768-dimensional text embedding; each passes through a single linear layer into a layer-normalized 512-dimensional space. LoRA adapters (rank 4, alpha 8, dropout 0.1) sit in the attention projections of all 32 Virchow2 blocks and all 12 BioMedBERT layers. Training used the symmetric InfoNCE objective from CLIP on all 653,209 image-caption pairs of the QUILT-1M corpus, with gradient caching to build 1,024 x 1,024 similarity matrices. QUILT-1M is not openly downloadable: its Zenodo record releases the files only on request, under research-use and no-redistribution terms. The model trained for 20 epochs (12,500 optimizer steps) with AdamW and cosine decay, and the reported checkpoint is the one with the lowest held-out loss.
Evaluation used the same prompts for every model, against six vision-language baselines including CONCH, KEEP and PathGen-CLIP. On nine public zero-shot patch benchmarks (among them LC25000, NCT-CRC, WSSS4LUAD, BACH, MHIST and SICAPv2), Lumen reached a mean chance-corrected balanced accuracy of 0.546 against 0.461 for the strongest baseline. For lymph-node metastasis detection it achieved an AUROC of 0.964 on 4,214 held-out internal Bern slides and 0.955 on 2,368 slides from nine external cohorts spanning six organs, including CAMELYON17 and HISTAI. On ARCH cross-modal retrieval it ranked third, behind CONCH and PathGen-L/14. A matched control that fully fine-tuned both encoders improved retrieval but gave no consistent gain elsewhere and lost ground on the external lymph-node cohorts.
Lumen suits pathology research groups that want to screen tissue for a concept without first building a labeled cohort: tissue typing, tumour detection, and triage of lymph-node slides for metastatic carcinoma across breast, colorectal, endometrial, lung, upper-gastrointestinal and melanoma cases. Its frozen image encoder also serves linear probes, which the authors report recover many of the fine-grained distinctions that zero-shot prompting handles poorly. The GitHub repository reproduces the benchmarks and exposes Lumen and a dozen baselines through one interface.
Lumen shows that a competitive pathology vision-language model can be assembled from existing foundation models by training well under one percent of their parameters, rather than by end-to-end retraining. Its comparison with QuiltNet, trained on the same QUILT-1M corpus yet scoring less than half of Lumen's patch benchmark result, points to the quality of the underlying encoders as a major driver of zero-shot performance. The alignment weights and code are released under CC BY-NC 4.0, but the assembled model inherits Virchow2's gated CC BY-NC-ND 4.0 terms, which restrict it to non-commercial research and forbid modifying the vision encoder. It has not been evaluated for clinical use, neither the internal Bern lymph-node cohorts nor the external lung cohort from University Hospital Cologne is public, and the work is a preprint awaiting peer review.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.