PMPRG

A real colorectal sign-out is not free prose. It is a fixed skeleton of fields — operation, diagnosis, depth of invasion, lymphatic invasion, perineural invasion, associated findings — and the skeleton changes when the specimen changes organ, so a kidney case asks a different set of questions than a colon case. It is also written once per patient, over however many slides the resection produced. Captioning architectures that compress a single whole-slide image into one vector and emit the whole report in one pass tend to answer only the questions that vector happens to encode, omitting fields the pathologist is required to fill.

PMPRG (Patient-level Multi-organ Pathology Report Generation) attacks that by writing the report field by field instead of sentence by sentence. It first classifies the specimen's organ from the pooled slide features, uses that prediction to retrieve the learned tag embeddings belonging to that organ from a tag-selection dictionary, and then treats each tag embedding as a query into the patient's regional features. The resulting tag-attended vectors are what the language model is conditioned on, one per report field, so every field the organ demands gets its own dedicated piece of visual evidence. It was developed by the High-performance Visual Computing Lab at Korea University with pathologists at Korea University Anam Hospital and at Seoul St. Mary's Hospital, The Catholic University of Korea, and was presented at MICCAI 2024.

The text backbone is a GPT-2 already pretrained on PubMed. Rather than fine-tune it, PMPRG replaces its self-attention layers with pseudo-self-attention and trains only the per-block projection parameters that inject the visual condition, leaving the language weights frozen — a deliberately small trainable surface for a clinical corpus that cannot be enlarged on demand.

#Key Features

• Patient-level input: regional representations from all of a patient's slides are concatenated before pooling, so a one-slide case and a twelve-slide case take the same path to a single report.
• Organ-conditioned tag dictionary: an organ classifier selects which set of learnable tag embeddings to retrieve, letting one model serve report templates whose fields differ by specimen type.
• Frozen medical text backbone: only the conditioning projections of the PubMed-pretrained GPT-2 are trained; at inference the model is fed a single <BOS> token and generates from the tag-attended features.
• Two-stage multi-scale encoder: MR-ViT builds a 1024×1024 region from one 1.25× magnification patch and fifteen 5× patches, encodes those sixteen tokens with a ViT over VGG16 patch features, and aggregates regions into a slide vector in a second stage — two DINO stages rather than three.
• Per-tag attention maps: the attention pooler exposes, for each report field separately, which regions of which slide the model attended to.

#Technical Details

Training used 7,422 whole-slide images from 1,991 patients collected at Korea University Anam Hospital under IRB approval, covering colon and kidney specimens, split 70/20/10 at random into train, validation and test. Stage one trains MR-ViT with DINO using HIPT's hyperparameters apart from input and output size; stage two runs 300 epochs and is selected on validation tag-classification accuracy, optimising a weighted sum of organ, tag and sentence cross-entropy losses (0.2, 0.6, 0.2). As a slide encoder, MR-ViT with VGG16 features reaches 0.8485 accuracy on diagnosis type against 0.8169 for ZoomMIL and 0.5455 for HIPT, while ZoomMIL's supervised end-to-end objective stays ahead on tumour grade (0.4688 versus 0.4242). On report generation, the full model scores METEOR 0.6834, ROUGE-L 0.6033 and BLEU-4 0.4654, with clinical-efficacy F1 0.5773 and accuracy 0.6022 across 26 classes; removing the tag classifier costs about 0.03 METEOR and ROUGE-L and about 0.05 BLEU.

#Applications

The intended use is drafting the structured report for a colon or kidney case from the patient's slide set, leaving the pathologist to verify and edit rather than compose. Because each generated field carries its own attention map, a reviewer can check the regions a particular claim was drawn from instead of accepting the text wholesale. The same tag-attended representations also support retrospective structuring of archived cases, where the fields have to be recovered from prose.

#Impact

The paper's substantive methodological argument is that report generation should be scored with clinical-efficacy metrics, not natural-language overlap alone. Its evidence base is narrow, however: every number comes from a random split of the single in-house Anam Hospital cohort that MR-ViT was itself pretrained on, with no external-cohort transfer, and the organ classifier and tag dictionary are sized to the two organs seen in training, so the model is not a general pan-organ reporter. The authors name three-scale input and additional organs as future work. The repository the paper gives as its code location holds only a stub README; neither training code nor weights have been released, and the Anam cohort is not public.