ViDRiP-LLaVA

Pathology vision-language models have been built almost entirely on still images. Systems such as PathAsst, PLIP, and PathChat learn from patch-caption pairs scraped from textbooks, publications, or social media, and answer questions about a frozen field of view. But diagnosis is not a static act. A pathologist moves across a slide, changes magnification, revisits a region, and narrates a chain of observations that converges on a sign-out. Educational histopathology videos on YouTube capture exactly that process, narration included, and had gone largely unused as a training signal for reasoning.

ViDRiP-LLaVA, from the QuIIL group at Korea University, is a large multimodal model built to consume that temporal narrative. It accepts three input regimes — a single patch image, an automatically segmented clip, and a manually segmented diagnostic video — and produces a detailed histological description followed by a definitive diagnosis, mirroring the descriptive-then-conclusive structure of a real sign-out. Its foundation is ViDRiP-Instruct, a dataset of 4,278 video and diagnosis-specific chain-of-thought instruction pairs distilled from educational pathology videos.

The scarcity of that high-quality video data is the central engineering problem the paper addresses, and it is solved by staging: knowledge is transferred first from abundant single-image pathology instruction sets, then from weakly annotated automatically segmented clips, and only finally from the small, carefully segmented diagnostic videos.

#Key Features

• Three input scenarios in one model: Single patches, automatically keyframe-extracted clips, and manually segmented diagnostic videos are all handled by the same checkpoint family, with separate image and video variants released.
• Chain-of-thought sign-out: Outputs pair a structured histological description — architecture, nuclear features, mitotic activity, stroma — with a final diagnosis, rather than emitting a bare label.
• Staged transfer from image to video: Four sequential training stages move from vision-language alignment through image instruction tuning and a mixed image-plus-clip stage to fine-tuning on segmented videos, compensating for the small video corpus.
• Curated video pipeline: 5,917 raw YouTube videos are temporally segmented two ways, filtered by tissue detection, OCR exclusion, and inpainting, then converted into instruction pairs by an LLM with a 0-5 quality gate.
• Non-commercial release: Both checkpoints and the training dataset are public, under licenses that permit academic research only.

#Technical Details

The architecture follows LLaVA-OneVision, substituting Qwen-2.5-7B as the language decoder and using a SigLIP vision encoder whose features are projected into the language model's embedding space. Non-proprietary baselines and ViDRiP-LLaVA alike process 32 frames at 384x384 resolution. Training proceeds in four stages: alignment on 723k Quilt-1M samples, 223k PathAsst samples, and 4k bladder pairs; image instruction tuning on 107k Quilt-LLaVA and 100k PathAsst samples; a mixed stage combining image instructions with the automatically segmented ClipPath-Instruct set; and a final video stage on ViDRiP-Instruct, with LoRA applied to the language model. Instruction pairs are generated with GPT-4o-mini under a two-stage prompting scheme that scores each source subtitle on a 0-5 scale and retains only those scoring 3 or above.

Evaluation uses the Video-ChatGPT protocol, scoring context, correctness, and detail orientation on a 0-5 scale with GPT-3.5-turbo-0613 as judge. ViDRiP-LLaVA reaches an average of 2.77 and a normalized score of 55.40, above GPT-4o at 2.58 and 51.60, Gemini-2.0-Flash at 2.21 and 44.10, Qwen2.5-VL at 2.02 and 40.30, and a LLaVA-OneVision baseline fine-tuned directly on the video set at 1.76 and 35.21. Removing the mixed image-plus-clip stage drops the average to 2.70. LoRA tuning of the language model edges out full fine-tuning, and training on half the video data still reaches 2.73.

#Applications

The model targets pathology education and research settings where diagnostic reasoning, not just classification, is the object of interest: generating narrated walkthroughs of slide regions, drafting descriptive findings for review, or serving as a baseline for work on temporal reasoning over histology. Because it accepts single patches as well as video, it can also be used in conventional patch-level question answering. The released ViDRiP-Instruct dataset and its test split give the field a reusable benchmark for pathology video reasoning, which is arguably the more durable contribution.

#Impact

ViDRiP-LLaVA establishes video as a viable training modality for pathology multimodal models and shows a 7B open model outscoring substantially larger proprietary systems on a domain benchmark — a result that depends heavily on domain-specific instruction tuning rather than scale. The limitations are stated plainly. Evaluation is confined to the authors' own YouTube-derived test set, scored by an LLM judge rather than by pathologists, so external validation is absent. Training data inherits whatever bias is present in educational video selection. And the licensing is restrictive: code and weights are CC BY-NC 3.0 and the dataset CC BY-NC-ND 3.0, with the authors explicitly barring commercial use and any application in clinical care or medical decision-making.