Loading…
Reference-guided anatomical segmentation for medical images, pairing vision-language spatial reasoning with a co-trained SAM 2 mask decoder.
No providers recorded yet. Browse all providers
Asked which coronary segment a catheter tip has reached, an interventional cardiologist does not answer by recognising that segment — under fluoroscopy one branch looks much like another. The answer comes from holding the frame against a labelled reference angiogram and reasoning about where the tip sits relative to neighbouring branches. RAU builds that habit into a model: rather than asking a network to know every structure a priori, it supplies an annotated reference image and asks the model to infer the target's anatomy from the spatial correspondence between the two.
RAU was developed at United Imaging Intelligence's Boston, Massachusetts lab and published at ECCV 2026. It takes a Qwen2.5-VL-7B-Instruct vision-language backbone through supervised fine-tuning and then Group Relative Policy Optimization (GRPO), training it to answer anatomy questions about an unlabelled target while a labelled reference is held alongside in the prompt. That reasoning is then wired down to pixels: the model emits a <SEG> token whose embedding an MLP adapter projects into the memory space of a SAM 2 decoder, which is co-trained rather than frozen and produces the final mask from the reference mask's memory tokens together with the language-derived prompt.
The framing matters because the usual alternative — pretraining a medical segmentation model on ever more expert-labelled data — runs straight into the annotation scarcity it is meant to relieve. Reference conditioning shifts the burden from corpus volume to a single labelled exemplar, and the trained checkpoint carries to modalities it never saw.
Training used 8 NVIDIA A100 80 GB GPUs under a QLoRA regime — 4-bit quantised base weights, bfloat16 compute, LoRA adapters of rank 8 — with supervision from RAOS, a whole-body CT collection of 413 patient scans annotated for 19 organs, and ARCADE, 1,500 X-ray coronary angiography frames labelled at branch and segment level. The segmentation loss combines binary cross-entropy and soft Dice; the GRPO reward adds format validity to segmentation quality.
On the question-answering stage alone, labelling accuracy on held-out RAOS CT rises from 16.62% for the untuned Qwen2.5-VL-7B to 64.11% after supervised fine-tuning and 70.68% after 2,400 GRPO steps. The full VLM-plus-SAM 2 system reaches 89.38% labelling accuracy on RAOS and 81.62% on ARCADE, and out of distribution 61.87% on LERA bone radiographs and 95.41% on CAMUS cardiac ultrasound. Segmentation Dice is 0.7151 on RAOS, 0.6754 on ARCADE, 0.7503 on CAMUS and 0.7010 on LERA, against 0.2435–0.4290 for a MedSAM2 memory baseline fine-tuned on the same reference setup and 0.0346–0.2592 for stock SAM 2. Accuracy depends on how close the retrieved reference is: stratified by LPIPS distance, RAOS labelling holds above 92% for well-matched pairs and falls to 61.39% for the most dissimilar.
The reference-based setting fits clinical tasks where a labelled prior image exists but per-case labels do not: naming the vessel segment a wire occupies during an intervention, propagating organ contours for radiotherapy planning, tracking a lesion across timepoints, and pre-labelling images so an annotator edits rather than draws. Because a new anatomy or modality needs only an annotated exemplar instead of a training run, groups that cannot assemble a labelled corpus can still apply the model to their own cohorts.
RAU treats anatomical understanding as a correspondence problem rather than a recognition problem, and its baselines argue the distinction is real: medically pretrained vision-language models such as MedVLM-R1, HuatuoGPT-Vision and Lingshu score close to the untuned backbone on the same reference-based questions, and reasoning-segmentation systems that do well in distribution degrade sharply outside it. The work is limited to 2D single-frame inputs, with temporal sequences and 3D volumes left as future work, and the paper carries no code-availability statement — as of September 2026 no repository or checkpoint has surfaced — so reuse presently means reimplementation from the paper.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.