Mem-GAN

Single-cell analysis of multiplexed fluorescence microscopy depends on two signals arriving cleanly: a nuclear stain and a membrane stain. In practice the membrane channel is the one that fails. Antibody concentration varies, epitopes are unevenly accessible, images drift out of focus, and in assays that measure RNA and protein in the same section the protein signal is weaker still. Tissue sections therefore routinely contain cells whose membranes are faint, partial, or absent, and segmentation tools such as Cellpose fall back on the nucleus alone — missing cells entirely, or drawing boundaries that stop at the nuclear edge.

Mem-GAN addresses that failure at the image level rather than the algorithm level. Instead of dilating nuclear masks or tessellating space to guess where a cell ends, it synthesizes the missing membrane channel outright, framing the task as unpaired image-to-image translation: take the appearance of cells with well-defined membrane staining and impose it on the content of cells that lack it. The reconstructed image — real nuclei preserved, pseudo-membranes added — is then passed to an off-the-shelf segmenter.

The model was developed by researchers at Michigan State University, Stanford University, NanoString Technologies, Jilin University, and MD Anderson Cancer Center, and released through the OmicsML collective. It was first posted as a preprint in November 2023 and accepted to the BioKDD 2025 workshop.

#Key Features

• Membrane synthesis from nuclei alone: Given only a nuclear channel, the model produces a membrane channel consistent with the cell's real extent, recovering cells a nucleus-only segmentation would merge or miss.
• Separate tumor and immune generators: Epithelial and tumor cells present jagged, heterogeneous membranes while lymphocytes are round and uniform, so two independent models are trained rather than one that averages both.
• Weak-membrane repair: A third model maps tumor cells with partial or faint signal onto the well-stained tumor distribution, completing boundaries from whatever real signal exists.
• Unpaired training: Cycle-consistent adversarial training removes the need for matched before-and-after images of the same cell, which do not exist for staining artifacts.
• Nucleus-preserving reconstruction: Post-processing merges the generated membrane channel back onto the original nuclear channel, so no measured nuclear signal is replaced by synthetic content.

#Technical Details

Mem-GAN comprises three separately trained CycleGAN models, each with two generators and two discriminators, optimized with an adversarial loss plus a mean-squared-error cycle-consistency term weighted at λ = 10. Training used the publicly released CosMx spatial molecular imaging dataset of formalin-fixed, paraffin-embedded non-small cell lung cancer tissue — eight samples, 120 fields of view, roughly 390,000 cells — of which four samples (Lung5_Rep1/2/3 and Lung6, 30–32 FOVs each) were used. Each FOV is 5,472 × 3,648 pixels at 0.18 µm per pixel and was cropped into 456 × 456 patches holding 10–50 cells; five trained annotators masked tumor and immune cells and filtered noisy staining to build the source and target sets. Stains were DAPI, CD298, PanCK, and CD3; membrane channels were combined and the nuclear channel subtracted to raise contrast. Training ran end-to-end in PyTorch on a single Nvidia RTX A6000 using Adam at a learning rate of 0.0002 for 100 epochs, decayed linearly to zero over 100 more.

Evaluation is indirect, through downstream segmentation. Taking Cellpose boundaries on the original membrane-stained images as ground truth, mask intersection-over-union from nuclei-only input rose from 0.2788 to 0.3084 on average across the four samples, with the largest gain on Lung6 (0.2839 to 0.3516). For weak-membrane repair, where no ground truth exists, five blinded domain experts scored 40 image pairs on a five-point scale: membrane completeness improved from 1.98 to 3.43, membrane quality from 2.28 to 3.13, and cell segmentation from 2.35 to 3.38.

#Applications

The intended user is anyone segmenting cells from multiplexed immunofluorescence in spatial transcriptomics workflows, where cell typing, neighborhood analysis, and transcript-to-cell assignment all inherit the segmentation's errors. Because Mem-GAN emits an image rather than a mask, it slots in ahead of an existing pipeline as a preprocessing step and leaves the choice of segmenter untouched. It is most useful for fields of view with uneven staining that would otherwise be discarded, and for panels where membrane markers were dropped to free channels for other targets.

#Impact

Mem-GAN reframes a segmentation problem as an image restoration problem. Its results are a proof of concept rather than a general-purpose model: all reported evaluation is on CosMx lung cancer tissue from two patients, and the authors state that the model degrades on out-of-distribution tissues and complex morphologies, with additional training data needed for new staining patterns. It can also be misled by spurious signal, generating membrane around free-floating markers that have no nucleus. The preprint has not been peer reviewed, and the MIT-licensed repository provides CycleGAN-derived training and inference code without pretrained checkpoints, so use requires retraining on the user's own curated patches.