Allen Institute for Cell Science
3D convolutional neural network that predicts fluorescent labels of subcellular structures from transmitted-light microscopy images, enabling label-free imaging of living cells.
pytorch_fnet (Fluorescence Prediction Network) is a 3D convolutional neural network developed by Chawin Ounkomol, Sharif Bhatt, Mary M. Maleckar, Forrest Collman, and Gregory R. Johnson at the Allen Institute for Cell Science. Published in Nature Methods in September 2018, the model introduced label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy as a practical approach to studying living cells without the photototoxicity, cost, and time requirements of fluorescent staining. By learning the relationship between transmitted-light morphology and fluorescent marker localization, the model enables researchers to generate virtual fluorescence images of subcellular structures from standard bright-field or differential interference contrast (DIC) microscopy.
The core biological challenge addressed by pytorch_fnet is the tension between observational power and cell viability in live-cell microscopy. Fluorescence microscopy provides extraordinary specificity — labeling individual organelles, cytoskeletal elements, or proteins with high signal-to-noise ratio — but the process of introducing fluorescent labels (whether through transfection, dye loading, or antibody staining) is often toxic, time-consuming, or incompatible with long-term live imaging. Transmitted-light imaging methods such as bright-field, phase contrast, and DIC, by contrast, are essentially benign to cells and can be performed indefinitely, but they produce images where organelles and protein structures are visible only as subtle variations in light intensity or phase that are difficult to interpret without training.
pytorch_fnet resolves this trade-off by learning to translate transmitted-light images into predicted fluorescence images, using paired training data where the same cells were imaged in both modalities. The model was trained on a large dataset of human induced pluripotent stem cells (hiPSC) from the Allen Cell Collection, where stable fluorescent cell lines expressing GFP-tagged markers for specific organelles were imaged by 3D transmitted-light and 3D fluorescence confocal microscopy. Because the same cell is imaged in both channels, the mapping between transmitted-light morphology and fluorescence localization can be learned directly from paired image stacks without any additional annotation.
pytorch_fnet uses a 3D U-Net architecture — a fully convolutional encoder-decoder network with skip connections — that has become the standard architecture for volumetric biomedical image translation tasks. The encoder downsamples the 3D input volume through four stages of 3D convolutions, batch normalization, and max-pooling, progressively extracting features at increasing scales and receptive field sizes. The decoder upsamples back to full resolution using transposed convolutions with skip connections from the corresponding encoder stages, preserving fine spatial details from the input that might otherwise be lost in the compressed bottleneck representation. The model was trained end-to-end using stochastic gradient descent with the Adam optimizer (learning rate 0.001) to minimize mean squared error between predicted and true fluorescence images over 50,000 mini-batch iterations per training run. Training data consisted of 3D image pairs from the Allen Cell Collection: for each of nine subcellular structures, a stable hiPSC cell line expressing a GFP-tagged protein marker was imaged by 3D transmitted light (either bright-field or DIC) and 3D confocal fluorescence microscopy. Data augmentation (rotation, flipping) was applied during training to improve generalization. In quantitative evaluation, model predictions achieved Pearson correlations with held-out ground truth fluorescence images ranging from approximately 0.61 (tight junctions) to 0.90 (nuclei), with the highest accuracy for structures that have the most distinctive and consistent transmitted-light morphological signatures. The model was implemented in PyTorch and all training code and datasets are available for download.
pytorch_fnet is directly applicable to any live-cell imaging study where researchers wish to monitor subcellular structure dynamics without the limitations of fluorescent labeling. For long-term time-lapse imaging — following a cell through division, differentiation, or in response to drug treatments over hours or days — label-free prediction eliminates photobleaching and phototoxicity that would terminate or perturb fluorescence imaging. In drug discovery, the model enables morphological profiling of compound effects on specific organelles from routine transmitted-light imaging, potentially increasing throughput by eliminating the staining step. For clinical applications in assisted reproduction and embryology, where fluorescent labeling is ethically and regulatorily restricted, label-free prediction of cellular structures from bright-field images of embryos represents a promising avenue for non-destructive quality assessment. The integrated multi-structure predictions also provide a form of virtual staining that can complement or replace labor-intensive immunofluorescence staining in histology applications when working with live cultures.
pytorch_fnet established label-free prediction of 3D fluorescence from transmitted light as a viable and broadly applicable paradigm, a concept that has since been extended by numerous groups to a wide range of imaging modalities, cell types, and biological structures. The Nature Methods paper has been highly influential in the computational microscopy community and is recognized as a foundational reference for virtual staining and content-aware image restoration approaches. It helped inspire a generation of related models including content-aware image restoration (CARE), Virtual Staining Networks, and in-silico labeling methods that apply similar paired-image training strategies. The Allen Institute's open release of both the trained models and the large-scale hiPSC training dataset has enabled broad adoption and independent benchmarking. An important limitation of the approach is that the predicted fluorescence images are statistical averages of what a fluorescent marker would show given the transmitted-light morphology, and cannot resolve stochastic spatial variation in protein localization that is biologically meaningful — for example, transient protein clustering events that vary from cell to cell without a consistent transmitted-light correlate.