Neuro-JEPA

A routine brain MRI exam is not one image. It is a stack of complementary acquisitions — T1-weighted for anatomy, T2-weighted for fluid sensitivity, FLAIR for the same fluid contrast with cerebrospinal fluid suppressed so that edema and white-matter lesions stand out. A radiologist reads them together, and the diagnosis often lives in the disagreement between them. Most neuroimaging models take a single sequence, and the multimodal ones are usually evaluated only in the convenient case where every sequence is present — which leaves the field unable to say whether jointly encoding contrasts helps, or whether the gain is simply more data at test time.

Neuro-JEPA, from NYU Grossman School of Medicine and NYU's Center for Data Science, is a single encoder trained to place all three core structural contrasts in one representation space. It extends the V-JEPA 2 joint-embedding predictive framework to volumetric MRI: rather than reconstructing masked voxels, the model predicts the latent embedding of masked 3D patches from the visible context, with a momentum teacher supplying the targets. The capacity needed to hold three physically distinct contrasts in one backbone comes from a sparse Mixture-of-Experts, so different anatomies and sequences can recruit different experts without the parameter count being paid on every forward pass.

The paper pairs the model with a methodological argument about how neuroimaging foundation models are judged. Compared against a supervised convolutional network trained from scratch for each task, the existing models the authors evaluated showed inconsistent gains — a result about the field's benchmarking conventions, and the basis for the authors' call to include simple baselines, clinically heterogeneous cohorts and controlled unimodal-versus-multimodal comparisons in evaluation protocols.

#Key Features

• One encoder across three contrasts: A single architecture handles T1w, T2w and FLAIR, and the same checkpoint is evaluated in unimodal, multimodal and cross-domain configurations rather than requiring a per-sequence model.
• Sparse Mixture-of-Experts backbone: MoE routing is enabled on alternating transformer layers, using 2 shared experts of 16 total, with 6 activated per token, so total capacity scales faster than compute.
• Foreground-aware latent prediction: Because skull stripping and defacing leave 40–60% of each volume as background, the standard JEPA L1 loss is reweighted to down-weight predicted background voxels, forcing the objective onto brain tissue.
• Multiscale masking for volumes: Masks are sampled across three spatial-scale ranges to a 0.75 ratio, replacing the large block masking that video JEPA uses and that obscures the fine detail small lesions depend on.
• Stabilized expert routing: Auxiliary-loss-free bias updates are modified with token-averaged error correction, zero-mean projection and bias clipping, preventing the expert collapse the authors observed otherwise.

#Technical Details

The backbone is a 3D vision transformer with 12 layers, 768 hidden dimensions, 3072-dimensional feedforward blocks, 12 attention heads and 3D rotary position embeddings. Volumes are resampled to 96 × 108 × 96 and tokenized with 12 × 12 × 12 patches, yielding 576 tokens per scan. The base configuration activates 86M of 122M total parameters; a large configuration activates 303M of 429M and adds a further 1.5–1.9% AUROC and 1.9–4.1% AUPRC. Pretraining used 1,551,862 scans (683,860 T1w, 402,192 T2w, 465,810 FLAIR) from 428,647 studies and 282,693 patients in the NYU Langone imaging archive, converging in under 200 epochs. Evaluation spans 47 downstream tasks across three health systems — NYU Langone, NYU Long Island and Massachusetts General Hospital — plus 12 public cohorts including ADNI, NACC, OASIS-3, PPMI, ABIDE, ADHD-200, UCSF-PDGM and OpenBHB. Against NeuroVFM, BrainIAC and VoCo, Neuro-JEPA reports mean gains of 4.4–6.4% AUROC and 6.4–9.4% AUPRC on unimodal tasks and 5.8–7.6% AUROC on multimodal ones, with few-shot AUROC improvements up to 15% on gliosis and multiple sclerosis.

#Applications

The evaluated endpoints are the ones structural MRI is already ordered for: stroke identification and classification, 90-day modified Rankin Scale outcome and hospital length of stay, hematoma subtyping, tumour characterization including glioblastoma and IDH mutation status, multiple sclerosis and gliosis detection, brain age regression, and neurodegenerative endpoints such as cerebral amyloid status and progression from mild cognitive impairment to Alzheimer's disease dementia. Because the encoder produces features usable with a linear or attentive probe, groups with small annotated cohorts can build task heads without pretraining of their own.

#Impact

Neuro-JEPA is an argument that pretraining design, not corpus size, is what decides whether a neuroimaging foundation model transfers: it reports outperforming NeuroVFM while using roughly a third of that model's pretraining data. Its scope is narrow — three structural sequences, with no claim over diffusion, perfusion, functional MRI or CT — and the evaluation is retrospective; the authors state the model has no regulatory clearance for clinical use. The work remains a preprint awaiting peer review. Code is released under MIT, while the weights are gated behind manual institutional approval and a non-commercial, non-derivative licence that forbids redistribution.