InstEditSeg

A segmentation network's usual output is a binary mask: a black-and-white stencil that looks nothing like the colonoscopy frame it came from. Stable Diffusion, pretrained on hundreds of millions of natural photographs, has never been asked to produce anything of the kind, so adapting it to segmentation normally means bolting a task-specific decoder onto its latents. InstEditSeg instead asks the generator to do something it already knows — edit a picture. Given the instruction "segment the polyp region using red," it returns the same endoscopic frame with the lesion painted red, and the mask is read back from the color. The output stays inside the distribution the backbone was trained on, so the natural-to-medical domain gap narrows.

The framework was developed by Ziquan Liu, Zhewei Zhu and Xuyang Shi at Southwest University of Science and Technology and posted to arXiv in September 2026. It sits between two lines of work: diffusion segmentation models such as MedSegDiff, SDSeg and TSLDseg still emit standalone binary masks under task-specific objectives, while promptable foundation models such as MedSAM need a point or box drawn per lesion, which is awkward for low-contrast borders. InstEditSeg needs neither, and adapts the same Stable Diffusion v1.5 backbone that RoentGen and MorphoDiff repurpose for chest radiographs and cell morphology. One polyp checkpoint, trained once on four pooled colonoscopy sets, then runs unchanged across the whole evaluation suite; the skin task is a separate run of the same pipeline.

#Key Features

• Segmentation as instruction-driven editing: The target is a color-coded overlay composited onto the input image, so no prediction head or mask parameterization is needed; the instruction alone selects the category.
• DINO Feature Guidance blocks: A frozen DINOv3 ViT-S/16 supplies patch tokens projected into a four-level pyramid aligned with the U-Net's decoder stages, fused by channel concatenation and a zero-initialized convolution that leaves the pretrained weights undisturbed at step zero.
• Dual-branch classifier-free guidance: The input image already fixes the spatial layout, so text guidance is needed only for task selection, cutting sampling from three U-Net forward passes per denoising step to two.
• Cross-domain robustness: Keeping the output near the natural-image prior is where the advantage concentrates — on held-out institutions and imaging hardware, not in-domain folds.
• Multi-lesion completeness: The editing target carries no location prior, so the generator reasons over every candidate region instead of biasing toward the dominant lesion.

#Technical Details

The backbone is Stable Diffusion v1.5 at 512×512, with the VAE and DINOv3 frozen and the diffusion U-Net and CLIP text encoder fine-tuned; six active DINOv3 layers feed the guidance pyramid. An auxiliary segmentation decoder supervised with Dice and cross-entropy at weight λ = 0.3 trains alongside the noise-prediction loss for 20,000 steps, AdamW at 1e-4 and 5e-7, batch size 8, on one NVIDIA A100 40 GB; inference is DDIM, 25 steps, guidance scale 7.5. The polyp model pools 1,967 training images from Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB and ETIS-LaribPolypDB, while PolypGen (1,411 images) is excluded from training entirely, as is ISIC2017 (600) for the skin model trained on ISIC2016. On unseen PolypGen it reaches the best Dice and IoU of every method compared, 83.92 and 77.50, and on unseen ISIC2017 the best Dice, 83.14; on the 119 multi-polyp PolypGen images it scores 75.93 Dice against 63.18 for the EMCAD baseline. In-domain it is competitive but not ahead — 92.10 Dice on Kvasir-SEG against EMCAD's 93.74 — and costs roughly 3.6 s per image against EMCAD's 218 ms.

#Applications

The immediate uses are polyp delineation in colonoscopy frames and lesion measurement in dermoscopy, both settings where a model is routinely deployed on scopes, lighting and patient populations it was not trained on. The instruction interface removes the per-lesion clicking that promptable segmenters require, which matters most for the faint, flat lesions that are hardest to box. The released code extends the same training path to a 14-class BTCV abdominal CT instruction set, giving groups one architecture to carry across modalities without writing a new decoder for each.

#Impact

InstEditSeg's contribution is a reframing rather than a leaderboard result, and the paper is unusually direct about where it pays. Rendering the annotation onto the image, instead of beside it, buys domain robustness and multi-lesion recall at the cost of inference latency. Two limitations are structural: the instruction selects a category but cannot parse spatial or size modifiers such as "segment only the largest polyp," and accuracy is sensitive to the color requested, with genuine failures on colors outside the training palette that recede only as the palette grows. The work is a preprint awaiting peer review, and while training and inference code is released under MIT, no trained checkpoint accompanies it — reproducing the reported numbers means rerunning the 20,000-step training from the public benchmark data.