EndoVLM

Gastrointestinal endoscopy generates enormous volumes of imagery, but almost all of it is unordered, highly redundant, and uncurated: a single examination yields dozens or hundreds of frames, most of them near-duplicates or non-diagnostic. Endoscopy foundation models have therefore been built almost entirely from self-supervision on images or video, discarding the clinical report that a gastroenterologist writes for every procedure. EndoVLM is a vision-language pretraining approach that recovers that supervision, treating the structured anatomical and pathological language of routine reports as the training signal for an endoscopic image encoder.

The central difficulty is a modality gap rather than a data shortage. A report describes what was seen at specific anatomical sites, but nothing in the record says which frames correspond to which description, and standard image-text contrastive training assumes a one-to-one pairing that endoscopy does not provide. EndoVLM addresses this with three mechanisms that jointly select the informative frames, align them to clinical taxonomy, and preserve pixel-level detail.

The model was described in an August 2026 preprint whose authorship is led from DAMO Academy, with collaborators at Zhejiang University, Shanghai Jiao Tong University, and affiliated hospitals. It sits alongside earlier endoscopy work in a different regime: Endo-FM learns from video without text, and EndoChat is a conversational assistant for surgical scenes, whereas EndoVLM is a report-supervised representation learner.

#Key Features

• Anatomy-Guided Sparse Pooling: Textual descriptions from the report serve as queries driving sparse attention over an image set, aggregating the semantically salient frames into anatomy-specific visual representations without relying on temporal ordering.
• Progressive Semantic-Aware Alignment: Clinical taxonomy — anatomical site and pathological status — is encoded as structured soft targets, moving training from coarse patient-level matching toward fine-grained localized alignment.
• Semantic-Concentrated Masked Autoencoder: Masked reconstruction is applied only to the semantically rich frames selected upstream, adding low-level textural and geometric precision as a complement to the high-level alignment objectives.
• Zero-shot anatomical recognition: The taxonomy covers 17 gastrointestinal regions, eight for gastroscopy and nine for colonoscopy, enabling site classification on unseen datasets without task-specific fine-tuning.

#Technical Details

The vision encoder is a ViT-B/16 and the language encoder is PubMedBERT, operating on images resized to 224x224. Pretraining used over 400K endoscopic examinations retrospectively collected from two medical centers; after filtering, the curated corpus comprises 348K examinations and more than 18.6M images, each examination pairing a clinical report with its image collection. The masked autoencoder branch masks 75% of patches. Training ran for 100 epochs on NVIDIA A800 GPUs at a total batch size of 96 with AdamW, a base learning rate of 1.5e-4, and weight decay of 0.05.

Downstream evaluation spans video polyp diagnosis on PolypDiag (97.3 F1), ulcerative colitis severity grading on LIMUC (94.5 AUC, 74.4 F1), and polyp segmentation on CVC-12k (86.4 Dice), Kvasir-SEG (91.9 Dice), and ClinicDB (93.1 Dice), with generalization tested on the unseen ColonDB and ETIS sets. Baselines include DINOv3, MAE, CLIP, BiomedCLIP, and Endo-FM, alongside task-specific segmentation models such as Polyp-PVT. In the zero-shot setting the paper reports near-perfect AUC on upper-GI anatomical recognition and an 18% AUC margin over BiomedCLIP on video disease diagnosis.

#Applications

The intended use is as a frozen or fine-tuned backbone for AI-assisted endoscopy: real-time quality control that confirms every anatomical landmark was photographed, polyp detection and segmentation, disease grading for inflammatory conditions, and retrospective analysis of archived procedure images. Because the representation is aligned to report language, it also supports classification of new endoscopic categories by prompting with text rather than assembling a labeled training set — useful for clinical groups that hold imagery but little annotation.

#Impact

EndoVLM makes the case that the report archive already sitting in endoscopy units is a usable pretraining signal, and that the obstacle is frame-to-text correspondence rather than data volume. The frame-selection and taxonomy-alignment recipe generalizes in principle to other procedural imaging where unordered image sets accompany a narrative record. Practical adoption is currently limited: the work is a preprint that has not been peer reviewed, the repository carries no license, the 348K-examination pretraining corpus cannot be released for patient privacy and ethics reasons — only a toy dataset is provided for pipeline verification — and pretrained weights are stated as forthcoming upon acceptance rather than available today.