Melanocytic Lesion Report VLM

Millions of melanocytic skin lesions cross a pathologist's microscope each year, and the great majority are ordinary moles. Diagnosing a common nevus takes seconds; writing the report that documents it takes considerably longer. That asymmetry is the target of this work: automating the drafting step for the high-volume, low-complexity end of dermatopathology, where the diagnostic decision is not the bottleneck.

Researchers at University Medical Center Utrecht and Eindhoven University of Technology built a vision-language model for exactly this domain, released as a preprint in February 2025. The authors did not give the model a name; it is described throughout the paper by what it does. The design follows the Contrastive Captioner (CoCa) framework, adapted for the scale of computational pathology: whole-slide images are far too large to feed a vision transformer directly, so tile features are extracted with a frozen pathology foundation model and then compressed by a Perceiver before reaching the language decoder.

The model operates at case level rather than slide level, which matters clinically — a melanocytic excision typically yields several slides, and the report describes the case, not any one section. That aligns it with case-level pathology vision-language models such as CONCH and TITAN, though those are pan-cancer models while this one is deliberately narrow.

#Key Features

• Case-level report generation: All slides belonging to a case are encoded jointly, producing a single report of the kind a pathologist would sign out rather than a per-slide caption.
• Perceiver aggregation: An asymmetric attention module iteratively distills the tile feature vectors of an entire case into 513 trainable embeddings, one reserved for the contrastive objective, which keeps memory bounded regardless of case size.
• Frozen pathology encoder: Tile features come from UNI, a pretrained histopathology foundation model that stays frozen, so training effort goes entirely into aggregation and language modeling.
• Zero-shot cross-modal retrieval: The contrastive half of CoCa yields a shared image-text space, supporting text-to-image and image-to-text search over the case archive without any task-specific training.
• Biomedical language decoder: BioGPT supplies a decoder already pretrained on biomedical text, so pathology vocabulary and report phrasing do not have to be learned from the training corpus alone.

#Technical Details

Whole-slide images are tessellated into 224×224 tiles and encoded with the frozen UNI encoder into 1024-dimensional feature vectors, which the Perceiver aggregates into the case representation. The language side is BioGPT — a 24-block decoder-only transformer with 347 million parameters and a 42,384-token biomedical vocabulary — split into a unimodal component (blocks 1–12) with an attention-pooling layer for contrastive training and a multimodal component (blocks 13–24) with cross-attention to the image embeddings. Language weights are largely frozen; only the word embedding layer, the attention-pooling layer, and the cross-attention layers are trained, alongside the randomly initialized Perceiver. Training and evaluation used 42,512 H&E-stained whole-slide images and 19,645 reports from 14,978 patients seen at UMC Utrecht between 2013 and 2020. In a blinded reader study over 50 test cases, an expert dermatopathologist scored reports on a 1–5 scale for accuracy and usability: on the 25 common nevi, model-generated reports scored 4.5 ± 0.8 against 4.6 ± 0.6 for pathologist-written ones, while across all 50 cases, including rarer subtypes, the model scored 3.7 ± 1.2. Retrieval showed the inverse pattern, working better on rare subtypes than on common nevi, whose reports are near-interchangeable.

#Applications

The workflow this fits is routine dermatopathology sign-out: the model drafts the report for a case, the pathologist reviews, corrects, and signs. Given that report quality for common nevi is already at parity with human-written reports and that common nevi make up most of the caseload, the time saved is concentrated exactly where the volume is. The retrieval capability serves a different need — finding precedent cases in an archive by describing them in text, or surfacing the reports associated with a visually similar lesion, which is useful for training, second opinions, and cohort assembly for research.

#Impact

The work is a concrete demonstration that report generation can reach clinical parity when the domain is narrowed enough, and its reader study is more rigorous than the automatic text metrics that dominate this literature. The limits are equally clear and stated by the authors: training and evaluation come from a single institution and a single lesion category, report quality degrades on rare subtypes where errors matter most, and the reader study used one pathologist over 50 cases. Code and model parameters are promised on acceptance but have not been released, so the results cannot yet be reproduced or the model applied elsewhere.