ChatEXAONEPath

Most multimodal language models for pathology are trained on image patches — small crops pulled from public question-answering datasets. That suits benchmark construction but not diagnosis: a pathologist reads a whole slide, and the morphology that settles a case is often spread across regions no single 256-pixel tile contains. Scaling a multimodal LLM to the whole-slide image (WSI) means confronting a gigapixel input that cannot be fed to a language model directly.

ChatEXAONEPath is LG AI Research's answer to that gap: a WSI-level multimodal LLM that takes an entire slide and produces free-text diagnostic answers. Rather than pretraining a new backbone, it composes two existing models. A frozen EXAONEPath patch encoder embeds every tile on the slide; a CLAM-style gated-attention aggregator pools those tiles into a single slide representation; a small vision projector maps that representation into the token space of LLaMA2-7B-Chat, which is then instruction-tuned with LoRA. The recipe is the same one behind PathChat — freeze a pathology vision foundation model, LoRA-tune a general chat LLM on top — and ChatEXAONEPath's authors say their implementation drew directly on PathChat's open-source code.

The distinguishing contribution is data. The team built a retrieval-augmented pipeline to convert 10,094 paired TCGA whole-slide images and their free-text pathology reports into instruction-tuning data, then evaluated the resulting model with an LLM-as-judge protocol scored against the original reports. The work was posted as an arXiv preprint in April 2025 and has not been accompanied by code, weights, a model card, or a hosted API.

#Key Features

• Whole-slide input, not patches: The model consumes every tile of a WSI through an attention-based aggregator, so its answers are conditioned on slide-level morphology rather than a single crop chosen in advance.
• Frozen pathology encoder, adapted LLM: Only the aggregator, projector, and LoRA adapters are trained; the vision backbone and the base LLaMA2 weights stay fixed, which keeps training cost to a handful of GPUs.
• Report-derived instruction data: A retrieval-augmented pipeline (RAIDER) pairs OCR'd TCGA pathology reports with a fixed set of question templates, using a vector store to retrieve the passages that actually answer each question.
• Pan-cancer scope: Training pairs span the diverse tumor types in TCGA, so one set of weights handles slides from many organs without per-cohort retraining.
• LLM-graded evaluation protocol: Answers are scored by LLaMA3.1-70B-Instruct against seven criteria — accuracy, relevance, completeness, clarity, appropriateness, consistency, and presentation — with chain-of-thought rationales.

#Technical Details

Slides are tiled into 256×256-pixel patches at 0.5 µm/px (20× magnification) and embedded by EXAONEPath, a ViT-based pathology encoder trained with self-supervised contrastive learning on roughly 2.8 million patches. Patch embeddings are aggregated by a gated attention transformer network in the CLAM family, and an attention-pooler projector with a Linear-GeLU-Linear head maps the 512-dimensional slide embedding into LLaMA2's 4096-dimensional token space. Training runs in two phases on four A100-40GB GPUs: vision-language alignment (LLM frozen, batch size 128, learning rate 2×10⁻³), then LoRA instruction tuning (rank 64, alpha 16, dropout 0.05, learning rate 2×10⁻⁵).

Two instruction datasets were built from the same 10,094 report pairs: Dataset-v1, 10,094 representative captions written by GPT-4o, and Dataset-v2, 69,544 pairs expanded with LLaMA3.1-70B via RAIDER. The split is 8,960 training and 1,134 held-out pairs. On that test set the acceptance rates are 54.4% for v1, 42.9% for v2, and 62.9% for v3 — the headline configuration, which reuses Dataset-v2 but drops Macenko stain normalization from the patch encoding step. That the larger augmented dataset alone hurt performance is one of the paper's more useful findings: expanding text without expanding the underlying slides skews the image-text alignment.

#Applications

The intended use is a diagnostic copilot that reads a whole slide and answers open-ended clinical questions — major diagnosis, tumor subtype, relevant morphological findings — in the register of a pathology report. Realistic near-term settings are report drafting from archived slides, retrospective cohort curation over TCGA-like collections, and pathology education, where a model that explains what it sees beats a bare classification score.

#Impact

ChatEXAONEPath is a preprint rather than a peer-reviewed system, and its practical reach is limited by what has been released: no code, no weights, no API, and no stated license. Its evaluation is also narrower than the landmark pathology vision-language studies it follows — a single TCGA-derived report-generation benchmark of 1,134 pairs, graded by an LLM judge that the authors themselves report to be unstable, applying overly strict constraints and occasionally producing incorrect justifications. What the work does establish clearly is that WSI-level instruction tuning is reachable by composing an existing pathology encoder with an existing 7B chat model, and that the quality of report-derived instruction data — not its raw volume — is what moves the result.