SlideChat

A diagnostic slide is around 100,000 × 100,000 pixels, and no language model reads it whole. The pathology assistants that preceded SlideChat resolved that by shrinking the problem in one of two directions, and the paper's own baselines show what each direction costs: given sampled patches from a slide, GPT-4o answers 57.91% of SlideBench-VQA (TCGA) correctly; given a 1024 × 1024 thumbnail of the same slide, it drops to 34.07%. Patches keep the cytology and lose the slide; the thumbnail keeps the slide and loses the cytology. A pathologist asked about tumour staging or margin status needs both at once.

SlideChat keeps both by putting a second encoder between the patches and the language model. The slide is tiled into 224 × 224 patches, each embedded independently by a frozen patch-level encoder; the resulting long token sequence — tens of thousands of tokens for a large slide — is then passed through a slide-level encoder whose sparse attention lets every patch token see the rest of the slide before a projector hands them to the LLM. The patch tokens the model finally reasons over are therefore contextualised, not isolated.

It was developed by Shanghai AI Laboratory with Xiamen University, East China Normal University, Stanford University and Monash University, released as a preprint in October 2024 and published at CVPR 2025. It sits alongside patch- and region-level assistants such as PathChat and PathAsst, and beside WSI-LLaVA, a contemporaneous slide-level assistant that reports its own numbers on SlideBench.

#Key Features

• Whole-slide input: The model consumes every patch of a gigapixel slide in one pass rather than a sampled crop or a downsampled overview, so questions about tissue architecture and cellular detail can be answered from the same input.
• The slide encoder is what generalises: Ablating it leaves in-distribution TCGA accuracy roughly unchanged but costs over 10 points on the out-of-cohort BCNB set — the cross-slide context is doing transfer work, not benchmark work.
• SlideInstruction: A TCGA-derived instruction corpus of 4,181 slide captions and 175,753 visual question-answer pairs from 3,294 patients, generated from cleaned pathology reports and organised into microscopy, diagnosis and clinical categories.
• SlideBench: An evaluation suite of 734 caption pairs, 7,827 TCGA VQA items across 13 tasks, and 7,247 breast-biopsy VQA items from 1,058 patients across 7 zero-shot tasks, with closed-set questions filtered by four LLMs and reviewed by five pathologists.
• Attributable answers: Averaging patch-token attention across layers and heads surfaces the five patches that most drove a response, which the authors show align with the morphological features the text names.

#Technical Details

SlideChat is a LLaVA-style stack: a frozen CONCH patch encoder, a LongNet slide-level encoder using dilated sparse attention, an MLP projector, and Qwen2.5-7B-Instruct as the language model. Training runs in two stages — cross-domain alignment on the 4.2K caption pairs with only the slide encoder and projector unfrozen, then visual instruction learning on the 176K VQA pairs with the slide encoder, projector and LLM all trainable. On SlideBench-VQA (TCGA) it reaches 81.17% overall, 13.47 points above the best baseline, against 67.70% for MedDr and 42.00% for LLaVA-Med on patch inputs; on the held-out BCNB cohort it reaches 54.15%, and on the externally proposed WSI-VQA set 60.18%. An LLM ablation found Qwen2.5-14B-Instruct 2.57 points better on TCGA than the 7B model that shipped, which was chosen for compute efficiency. The repository later expanded SlideBench-VQA (TCGA) to 31 further cancer types and reports 74.12% on that larger set.

#Applications

The immediate uses are conversational: drafting a slide-level description for a case, asking follow-up questions about grading, staging or lymphovascular invasion, and answering biomarker-status questions of the kind the BCNB tasks cover (ER, PR, HER2). Because the answer is grounded in retrievable patches, it can also point a pathologist at the fields worth a closer look. For researchers, SlideInstruction and SlideBench are reusable in their own right — the first sizeable public instruction corpus at whole-slide resolution, and a benchmark other slide-level assistants now report against.

#Impact

SlideChat is the first openly released vision-language assistant operating at whole-slide resolution, and releasing the model, the instruction data and the benchmark together gave the field a common target where previously each group evaluated on private data. The limits are real: the training captions and questions are GPT-4 rewrites of TCGA reports covering ten cancer types, so supervision is report-level rather than pixel-level and the disease coverage is narrow; 54.15% on an out-of-cohort breast-biopsy set is well short of clinical reliability; and the model has not been evaluated prospectively or cleared for diagnostic use. Code is Apache-2.0 and the datasets are openly downloadable, while the weights require agreeing to share contact information before download.