OmniPath

A whole slide image routinely exceeds 50,000 x 50,000 pixels, while the vision encoders inside general large vision-language models accept a few hundred pixels on a side. Pathology assistants have coped by choosing one of two lossy options: feed a single high-magnification patch and lose the global context that determines a diagnosis, or feed a downsampled thumbnail of the whole slide and lose the nuclear and cellular features that determine a grade. Attention analysis makes the cost visible — in existing pathology assistants, the handful of image tokens the language model actually attends to sit outside the cancerous region a pathologist has outlined.

OmniPath, from Zhejiang University with the First Affiliated Hospital of its medical school, addresses both failures at once. Two strategies do the work: mixed task-guided feature enhancement, which trains the visual pathway on detection and segmentation so it encodes local structure rather than a single global summary, and prompt-guided detail feature completion, which uses the model's own attention over a thumbnail to pick the regions worth re-reading at full resolution.

This entry covers the pathology vision-language model. It is unrelated to OmniPath, the database of intra- and intercellular signalling knowledge maintained by the Saez-Rodriguez group, which shares the name but is a curated resource rather than a model.

#Key Features

• Prompt-guided detail completion: The top-ranked image regions by attention are retrieved from the original slide at full resolution and appended as supplementary input, with average pooling keeping the extra token count small enough not to slow inference.
• Detection and segmentation in dialogue: A <mask> token added to the language model's vocabulary, plus a ResNet-18 mask encoder and a SAM-style decoder, let the model return segmentation masks and bounding boxes as part of an answer.
• Multi-scale fusion vision encoder: Higher-resolution inputs are divided into tiles at the encoder's native size, encoded separately, reassembled by spatial position and concatenated along the channel dimension, so detail increases without inflating the token budget.
• Instruction data across three levels: Training tasks span tissue (cancer region and lymph node detection), structure (vessels, bile ducts, nerves, microvascular and neural invasion) and cellular scales (nucleus classification, and inferential tasks such as finding cancer cell nuclei inside a vessel).

#Technical Details

OmniPath is built on LLaVA-1.5, with the original CLIP ViT-L/336px vision encoder replaced by SigLIP ViT-SO at 384px and UNI added as an auxiliary pathology encoder, joined to the language model by a two-layer MLP projector with GELU activation. Unlike the two-stage recipe typical of vision-language assistants, all modules train together in a single stage: two epochs on eight A100 GPUs with AdamW, a 2e-5 learning rate and a global batch size of 128, over roughly 490K samples spanning 21 organs.

On slide-level diagnostic tasks the model reports 98.40% accuracy for hepatocellular carcinoma subtyping, 99.08% for intrahepatic cholangiocarcinoma grading and 98.72% for lung cancer subtyping, against 89.74%, 93.58% and 91.73% for a Quilt-LLaVA baseline fine-tuned on the same data. Zero-shot slide-level accuracy reaches 79.15% on PANDA and 59.33% on CAMELYON17. For referring detection it reaches an F1 of 92.13 on hepatocellular carcinoma regions versus 84.64, and segmentation Dice of 95.51 versus 84.51. Removing the detail-completion strategy costs 21.1 accuracy points on average.

#Applications

The target setting is interactive auxiliary diagnosis: a pathologist asks about a slide in natural language and receives a graded, subtyped answer together with the outlined region that supports it. The instruction set was assembled from diagnostic items in real pathology reports across several institutions, so the task list — grading, subtyping, microsatellite instability detection, tumour-infiltrating lymphocyte recognition, lymph node metastasis detection — tracks what appears on a reporting worksheet.

#Impact

OmniPath's contribution is architectural rather than one of scale: it shows that a pathology assistant can keep thumbnail-level speed while recovering the fine detail that resolution limits normally discard, and that adding dense visual tasks to the training mix shifts the language model's attention onto diagnostically relevant tissue. The authors are candid about the limits — medical knowledge depth is constrained by a training corpus dominated by image-caption pairs, zero-shot generalisation still trails supervised accuracy, and reasoning is not at the level of independent diagnosis. Neither code, weights, nor the 490K-sample training corpus have been released, and no license is stated, so the reported results cannot currently be reproduced independently. The paper remains a preprint under review.