LO-VLM

Optical coherence tomography (OCT) is the standard modality for non-invasive retinal imaging, resolving individual retinal layers in vivo at micrometer scale. Reading those volumes by hand is slow and inconsistent between observers, and the deep learning tools built to help have mostly returned a categorical diagnosis or a segmentation boundary — outputs that say what a scan is without saying what the reader is looking at. Vision-language models can close that gap by emitting a narrative, but the ophthalmic VLMs available before LO-VLM carried heavy compute requirements and covered a narrow slice of retinal pathology.

LO-VLM (Layer-wise OCT Vision-Language Model) is a 247M-parameter vision-language model built by the Alam lab in the Department of Electrical and Computer Engineering at the University of North Carolina at Charlotte, with retina specialists at Stanford University, the University of Illinois Chicago, and Wake Forest University School of Medicine. Its central idea is anatomical supervision: rather than pairing each B-scan with a free-text impression, the training captions are decomposed into five named retinal layers, so the model learns where in the retina a finding sits, not only that it is present.

The result sits between two families already in the catalog. Retinal foundation models such as RETFound learn general-purpose image representations but do not write text; broad medical VLMs such as LLaVA-Med write text but were not trained on OCT layer anatomy. LO-VLM is deliberately domain-specific, and small enough to run where a 3-billion-parameter general model would not.

#Key Features

• Layer-wise anatomical supervision: Each training caption is structured as <LayerName>: clinical summary, covering the nerve fiber, ganglion cell, inner plexiform, outer plexiform, and IS/OS layers, injecting anatomical priors directly into the learning objective.
• Two outputs from one checkpoint: The same fixed model performs six-class disease classification and free-form narrative generation on a new B-scan without per-case retraining.
• Compact by design: At 247M parameters, it outperformed a 3B-parameter general-purpose VLM that produced only generic captions on the same scans.
• Blinded specialist evaluation: Three board-certified retina specialists scored the generated summaries, a stronger test of clinical usefulness than text-similarity metrics alone.
• Partially released training corpus: The 27,000 public image-text pairs are published as a Hugging Face dataset; the institutional cohorts remain private.

#Technical Details

The architecture is an encoder-decoder derived from the BLIP framework: a ViT image encoder converts the B-scan into patch tokens, and a transformer text encoder-decoder generates language conditioned on that embedding. Three objectives are trained jointly and reframed for the layer-wise setting — retina layer-text matching, retina-layer contrastive alignment, and layer-conditioned language modeling. The multimodal corpus comprises 40,000 OCT B-scans paired with expert-validated summaries across diabetic macular edema, diabetic retinopathy, geographic atrophy, drusen, choroidal neovascularization, and healthy retina, assembled from 27,000 scans in the public Kermany dataset, 7,200 from an institutional Illinois collection, and 5,700 from a Wake Forest cohort. Training used 39,000 pairs with 1,000 held out, a batch size of 32, 50 epochs, and a 256-token caption limit.

On the held-out split, LO-VLM reached an SBERT similarity of 0.803 and a BERTScore F1 of 0.715, improvements of 8.2% and 28.8% over specialized VLM baselines, and 96% accuracy with an F1 of 96% on the six-class task — 13 percentage points above a ViT-Base classifier trained on the same images. In the blinded review, LO-VLM narratives averaged 8.5 out of 10 (SD 1.15) against 5.5 (SD 1.13) for RetinaVLM (p < 0.0001).

#Applications

The model suits workflows where an OCT scan needs both a triage label and a readable account of what was seen: pre-screening scan queues, drafting layer-level descriptions for the medical record, and decision support for optometrists and general practitioners in regions with few retina specialists. Because the outputs are structured by retinal layer, they also serve research groups annotating large OCT cohorts consistently.

#Impact

Published in Communications Medicine after preprinting on bioRxiv, LO-VLM makes the case that in a narrow imaging domain, anatomically structured supervision buys more than parameter count. Code is released under the MIT license and archived on Zenodo, and weights are on Hugging Face, though neither the model repository nor the dataset repository carries a written card. The scope is genuinely bounded: six conditions, 2D B-scans, and a single held-out split of 1,000 scans, with specialist scoring from three readers. Performance on scanner types, populations, or pathologies outside the assembled corpus has not been established, and the model has not been evaluated prospectively in a clinic.