MR-JEPA

Ejection fraction is a difference between two moments in the cardiac cycle, and myocardial strain is deformation measured across all of them. Both exist only in time. Yet most deep learning on cardiovascular magnetic resonance (CMR) reads one 2D frame at a time — a cine series arrives as a stack of independent pictures rather than a beating heart, a late gadolinium enhancement (LGE) stack as unrelated slices rather than a walk through the myocardium. The sequential structure that carries the measurement is discarded before the network sees it.

MR-JEPA treats a CMR study as video. It adapts LeJEPA — a joint-embedding self-supervised objective that trains a single shared encoder to agree across augmented views, with a Sketched Isotropic Gaussian Regularization term preventing collapse in place of the momentum teachers, stop-gradients and predictor networks that earlier JEPA variants need — to spatiotemporal medical input. Three changes make that work: the 2D patch embedding becomes a 3D tubelet embedding spanning several frames and a spatial patch; local crops are spatiotemporally masked so the encoder must infer from incomplete clips; and the encoder is initialized from a 2D CMR foundation model trained with the DINOv3 objective, with patch embeddings inflated along the temporal axis and positional encodings replaced by sinusoidal spatiotemporal ones. Pretraining runs once, without annotations, over 160,172 clips from 10,505 patients at two centers.

It comes from the Digital Technology and Innovation group at Siemens Healthineers in Princeton with the Technical University of Munich and Imperial College London, posted to arXiv in August 2026 and accepted at STACOM 2026, a peer-reviewed MICCAI workshop. It is the video counterpart to CMR-SSL, the same group's 2D multi-sequence CMR encoder, and where prior CMR video models confine themselves to cine, MR-JEPA pretrains on cine, LGE and parametric mapping together.

#Key Features

• One frozen encoder, six downstream tasks: Left and right ventricular ejection fraction, global longitudinal, circumferential and radial strain, and four-class disease classification all reuse the same checkpoint; only the task head is trained.
• Gated attention across views: Each view's [CLS] embedding is projected to 512 dimensions and aggregated by parallel Tanh and Sigmoid pathways whose product yields attention logits, so a task can weight the 2CH, 4CH, short-axis and LGE inputs differently.
• Spatial stacks and temporal videos in one space: An LGE stack covering the heart and a cine loop are both fed as video, so one representation covers dynamics and anatomy.
• Strain is where the gain concentrates: Against the strongest baseline, mean absolute error falls 15–27% across the three strain measures, with correlations of 0.726–0.805.

#Technical Details

The encoder is a 27.0M-parameter ViT-S/16 adapted for eight-frame grayscale clips, trained for 300 epochs with AdamW at learning rate 5×10⁻⁴, batch size 64 across four GPUs in bfloat16, using two global 224×224 crops and six local 96×96 crops per sample, in two stages of increasing spatial resolution. Pretraining data were acquired between 2013 and 2022 on 1.5T and 3T Siemens scanners and are 77% cine, 23% LGE and 0.4% mapping. On the Kaggle Second Annual Data Science Bowl cohort (n=427), left ventricular ejection fraction reaches an MAE of 4.79% with r=0.764 and a Bland–Altman bias of 0.22%, against 6.07% for the domain-specific CMR baseline and 6.70% for a natural-video model; right ventricular ejection fraction reaches 8.67% MAE, and the strain MAEs are 1.87 (GLS), 2.39 (GCS) and 5.33 (GRS). Four-class disease classification reaches a macro AUC of 0.868, behind the 0.882 of a baseline pretrained on 293K clips with clinical report supervision but ahead of the 0.741 of the natural-video model. Ablations are stark: dropping either the 2D initialization or the spatiotemporal masking sends LGE scar linear probing from 0.850 accuracy to chance.

#Applications

The target is the quantitative half of a CMR report — ventricular ejection fractions and myocardial strain, which today rely on chamber segmentation pipelines or expert contouring — plus triage toward dilated cardiomyopathy, hypertrophic cardiomyopathy and ischemic heart disease. Because the encoder stays frozen, adding a seventh measurement trains only a small attention head.

#Impact

The result worth carrying forward is that a plain self-supervised objective on video, with no text supervision and a fifth of the pretraining clips, matches or beats an encoder trained against paired clinical reports on every continuous measurement — which matters because report-image pairs are the scarce resource in medical imaging. The caveats the authors state are real. Pretraining used single-vendor data; apart from the ejection fraction task, downstream evaluation was on data from a pretraining center; ground truth for right ventricular ejection fraction and strain came from a previously validated automated pipeline rather than manual expert annotation; and mapping data are scarce and single-center. Neither code nor weights have been released, and the paper carries no code or data availability statement.