SCOUT

A pathologist does not read a slide in one direction. A suspicion about tumor grade changes which fields of view matter, and what those fields show revises the suspicion. Report-generation models have mostly not worked that way: patch embeddings are extracted once from a frozen encoder, and whatever context the model has arrives only after the visual representation is fixed. Context can then re-weight the evidence, but not change how it was read.

SCOUT — Semantic Context-aware mOdality fUsion Transformer, from Stony Brook University — makes the conditioning recursive instead. It carries three input streams for each whole-slide image: patch-level morphology from a frozen CONCHv1.5 encoder, a global slide embedding built from TITAN and GECKO deep features, and a concept vector scoring an expert-curated vocabulary of diagnostic findings. At every encoder layer the current patch representation is modulated by the slide and concept states through FiLM conditioning, and the modulated representations are pooled straight back to update those states for the next layer. Backbone and context co-evolve across depth rather than one being injected into the other.

The decoder keeps the three streams apart until the last possible moment. Each layer runs separate cross-attention over the patch, slide-conditioned and concept-conditioned representations, then combines them with a gate learned per token, fusion head and feature dimension. Those gate values are retained, so the model reports how much of each generated phrase drew on local morphology, whole-slide architecture or named diagnostic concepts — a description of its own internal allocation, not a causal explanation.

#Key Features

• Recursively updated context: Slide and concept states are recomputed after every encoder layer from the patch features conditioned on them, so contextual signals adapt as the visual representation is transformed.
• Concepts as an active conditioning stream: Findings such as cribriform pattern, comedo necrosis or perineural invasion enter as a conditioning input to the encoder, not as an auxiliary loss or a post-hoc annotation layer.
• Feature-wise gated fusion: A temperature-scaled softmax over the three streams is predicted for every token, head and within-head dimension, letting reliance shift between cellular description and diagnostic summary within one sentence.
• Offline feature pipeline: Patch, slide and concept features are precomputed once per slide through TRIDENT and stored as HDF5, confining train-time cost to the adaptation and fusion modules.

#Technical Details

Slides are tiled into non-overlapping 512x512 patches at 20x magnification and encoded to 768-dimensional CONCHv1.5 features; MLP adapters project patch, slide and concept inputs into a shared 768-dimensional space. Encoder and decoder each use four Transformer layers with four attention heads and dropout 0.2, and a Position-Aware Module applies 13x13, 7x7 and 3x3 depth-wise convolutions after each self-attention block. Each stream also learns softmax-normalized weights over encoder depth. Training uses AdamW at a learning rate of 7e-5 on four NVIDIA H200 GPUs, optimizing report likelihood plus a gate-entropy term that encourages selective modality use. Evaluation covers three cohorts totaling more than 20,900 WSI–report pairs and over seven cancer types: the TCGA-BRCA subset of PathText (1,024 slides), HistAI (12,476) and the MICCAI 2025 REG-2025 challenge set (7,405). Against WSI-Caption, HistGen and Bi-Gen trained on identical splits and features, SCOUT raised BLEU-4 from 0.1394 to 0.1561 on TCGA-BRCA and from 0.7355 to 0.7797 on REG-2025, and lifted the Clinical Report Quality Score on REG-2025 from 0.6985 to 0.7376. Bi-Gen retained a higher ROUGE-L on HistAI, so the gains are not uniform. REG-2025 ablations credit both halves of the design: adding the slide and concept streams by plain concatenation moved BLEU-4 from 0.7281 to 0.7554, FiLM conditioning to 0.7614, gated fusion to 0.7797.

#Applications

The target use is drafting a diagnostic narrative for a scanned H&E slide, across the breast, prostate, colorectal, gastric, lung, cervical and bladder cases the concept vocabulary covers. Retained gate values and per-stream attention maps make each draft inspectable: a reviewer can see whether a statement tracked cellular morphology or a concept prior before accepting it. The vocabulary is fixed before test time and draws nothing from target reports, so reaching a new organ means curating concepts rather than re-architecting the model.

#Impact

SCOUT is a preprint awaiting peer review; the authors point to a project repository holding source code, configuration files and the fixed splits, but it is not yet publicly accessible, and no trained weights have been released. Its contribution is methodological — evidence that report generation improves when domain knowledge conditions visual representation learning throughout the encoder instead of being fused at the end. Models were trained and evaluated within each cohort rather than across them, so robustness under cross-institutional shift is untested, and reports were scored by automated metrics rather than by pathologists. The LLM-assisted, expert-reviewed curation of the vocabulary is both mechanism and constraint: gaps in its coverage propagate straight into the conditioning signal.