FOCUS-3D

A light-sheet recording of a zebrafish embryo is one continuous object, but most cell segmentation tools treat it as a pile of unrelated pictures: each z-plane is segmented in 2D and the fragments are stitched back together afterwards. A boundary that is faint in one plane and sharp in the next gets resolved inconsistently, and cells that touch across a single slice are merged. FOCUS-3D removes the stitching step by predicting instance masks directly in the volume, so the axial dimension carries evidence rather than acting as an index into a stack.

FOCUS-3D was developed by Guoqiang Yu's group at Tsinghua University, with collaborators at the Biozentrum of the University of Basel, and posted to bioRxiv in August 2026. Its central design choice is to spend supervision where it is cheapest. Expert-curated 3D annotations are extraordinarily expensive, so the model is trained in three tiers of decreasing quantity and increasing quality: self-supervised masked-autoencoder pretraining on unlabeled volumes, then instance training on machine-generated "silver-standard" masks, then refinement on expert "gold-standard" annotations.

The result is a generalist rather than a per-dataset model. Where earlier volumetric workflows built on Cellpose or StarDist usually required retraining for each new specimen, FOCUS-3D is evaluated as a single checkpoint across species, tissues, fluorescent reporters and imaging modalities.

#Key Features

• Native volumetric instance segmentation: Object queries produce full 3D masks in one pass, avoiding the slice-by-slice inference and stitching heuristics of existing 3D pipelines and roughly tripling throughput relative to Cellpose on a 12-GB VRAM budget.
• Three-tier training curriculum: Self-supervised pretraining, then large-scale silver-standard supervision, then gold-standard refinement — each stage measurably contributes, with silver-standard training alone reaching most of the final accuracy.
• Anisotropy-aware feature extraction: The adapter's early pyramid stages skip downsampling along Z, preserving axial detail in the anisotropic volumes that light-sheet and confocal acquisition routinely produce.
• Nuclei and membrane checkpoints: Alongside a general checkpoint, separate weights are released for nuclear and membrane labeling, the two dominant reporter strategies in developmental imaging.
• Human-in-the-loop napari plugin: An integrated interface supports block-wise manual curation, one-click correction of individual cells, fine-tuning on the corrected patches, re-segmentation, and downstream morphometry and contact-graph analysis.

#Technical Details

The architecture pairs a 3D masked autoencoder backbone with a query-based mask decoder in the style of Mask2Former. The encoder is ViT-Large scale — 24 transformer layers, embedding dimension 1008, 16 attention heads — operating on 32x96x96 voxel inputs with 8x12x12 patches. An anisotropy-aware ViT-Adapter converts its single-scale features into a hierarchy, a 3D deformable-attention pixel decoder produces mask features, and a masked transformer decoder with 300 object queries and 9 layers emits the instance masks; 100 auxiliary denoising queries stabilize query learning.

Pretraining used roughly 5.1 TB of raw 3D microscopy from 27 public and 20 in-house datasets, for 400 MAE epochs. Silver-standard training covered 19,583,728 machine-generated instances, and refinement used a gold-standard pool of 137 volumes and 466,343 curated instances across seven species. Ablations isolate each stage: random initialization drops AP@0.5 from 0.583 to 0.428, and inserting silver-standard training before gold-standard refinement raises AP@0.5 from 0.583 to 0.785. As a generalist without target-dataset fine-tuning, FOCUS-3D reaches AP@0.5 of 0.575 on a Drosophila benchmark against 0.342 for Cellpose and 0.218 for StarDist, and on an Arabidopsis ovule dataset absent from both pretraining and fine-tuning it scores 0.830 zero-shot. On 2D cross-sections it also exceeds CellSAM across zebrafish, Drosophila and C. elegans.

#Applications

The model targets developmental and cell biologists who image whole embryos, organoids, spheroids or plant tissue in 3D and need per-cell shape, volume and neighbor relationships rather than counts. Because it runs inside napari with pretrained weights on Hugging Face, a wet-lab group can segment a volume, correct the errors that matter for their specimen, fine-tune on those corrections, and compute morphometry without leaving the viewer. Accurate volumetric masks also feed downstream tracking and lineage reconstruction.

#Impact

Applied to dual-channel live imaging of zebrafish notochord morphogenesis, FOCUS-3D resolved three successive phases — compaction, consolidation and deformation — that the authors then mapped onto transcriptional states in an independent spatial transcriptomics dataset, an analysis that depends on membrane-resolved single-cell shape over time. The code is released under BSD-3-Clause and the three checkpoints under Apache 2.0, but the gold-standard annotation resource the paper presents as a major contribution is described only as being prepared for release, so the training data cannot yet be inspected. The authors are candid about the remaining failure modes: weak or discontinuous boundaries, intensity heterogeneity, densely touching cells and extreme object sizes still cause errors, dataset-specific fine-tuning may be needed when new data depart sharply from the training distribution, and the framework covers nuclei and membrane-defined cells rather than organelles. The work has not yet been peer reviewed.