PhysioJEPA

An ICU bedside monitor streams arterial blood pressure, ECG lead II and photoplethysmography at 125 Hz around the clock, and much of what it records is not physiology: electrode drift, motion artifact, a clinician handling the arterial line. A model trained to reconstruct those raw samples spends capacity reproducing exactly the part an experienced eye throws away. PhysioJEPA avoids reconstruction entirely. A context encoder sees a masked subset of one-second patches, a predictor estimates the embeddings of patches it never saw, and an exponential moving average copy of the encoder produces those targets from the full segment. The loss compares predictions to embeddings rather than waveforms, so high-frequency noise the target encoder has already discarded cannot become a training signal.

The model comes from the Windreich Department of Artificial Intelligence and Human Health at Icahn School of Medicine at Mount Sinai and was presented at the Fifth Machine Learning for Health Symposium. Joint embedding predictive architectures had reached physiological signals before — ECG-JEPA operates on short, single-modality 12-lead ECG windows — but not the long, multi-channel streams bedside monitors actually produce. PhysioJEPA extends the framework to 30-minute, three-channel segments and adds what multi-channel input requires: a learnable mask token and a depthwise convolutional tokenizer per channel.

The authors' claim is that one pretrained encoder, left frozen, answers more than one clinical question. Non-linear probes on those frozen representations estimate five-minute-ahead risk of hypotension and elevated shock index, and the MIMIC-III-trained stack then scores on a separate health system's monitors with no retraining.

#Key Features

• Embedding-space prediction: Targets come from the EMA target encoder's representations, not raw samples, keeping the objective away from sensor noise a reconstruction loss is obliged to model.
• Channel-specific mask tokens: Learnable tokens are repeated per target position per channel and shuffled with context tokens, so the predictor knows which channel it must fill in.
• Depthwise convolutional tokenization: Each channel gets its own kernels over 1-second, 125-sample patches, yielding 1,800 tokens per 30-minute segment and preserving per-channel structure before attention mixes it.
• Frozen encoder, lightweight probe: An attentive classifier pools the encoder output through a single learned query vector; encoder weights never move during probing, so one pretraining run serves both outcomes.
• Cross-site transfer without retraining: The MIMIC-III-trained encoder and classifier are applied unchanged to Mount Sinai Bedmaster recordings from six adult ICUs.

#Technical Details

Context and target encoders are 3-layer PatchTST transformers with 8 attention heads, dimension 512 and feedforward size 2,048; the predictor is a smaller 2-layer, 4-head, dimension-256 PatchTST; rotary positional embeddings encode patch position. Target masks select 10–30% of patches and context masks 10–40% of the remainder; training ran 100 epochs with AdamW and a one-cycle schedule on two H100 GPUs. Pretraining used the MIMIC-III Waveform Database Matched Subset — 356,903 thirty-minute segments totalling 10,707,090 minutes, from 4,282 ICU stays and 2,631 patients, after 1,378 of 5,660 eligible stays were dropped for 20% or more constant or missing values. On the held-out MIMIC-III test set PhysioJEPA reaches AUROC 0.833 for hypotension and 0.954 for shock index, against 0.867 and 0.956 for a PatchTST encoder trained by masked autoregression on the same data, 0.729 and 0.923 for ECG-JEPA, and 0.778 and 0.950 for a supervised convolutional baseline. On 100 external Mount Sinai stays per task it scores 0.781 and 0.923, ahead of PatchTST (0.761, 0.873), ECG-JEPA (0.568, 0.888) and the supervised model (0.695, 0.782). The paper reports no total parameter count.

#Applications

The immediate use is early warning in the ICU: a five-minute horizon leaves room to act before a hypotensive episode or a rising shock index becomes an emergency, and because it reads the monitor stream directly it needs no electronic health record integration, so it ports across units with different charting systems. The frozen, task-agnostic representation is the broader offer — outcomes such as sepsis onset or respiratory failure call for fitting another probe, not another pretraining run. Deployment is bounded by the ABP channel, which requires an invasive arterial line that is not in place for every monitored patient or available at every institution.

#Impact

PhysioJEPA follows PFTSleep, the same group's representation model for overnight polysomnography, and carries joint embedding prediction from sleep studies and single-modality cardiac work into continuous critical care monitoring. The result worth taking forward is the external validation: PatchTST edges it on the internal test set, but the ordering reverses on unseen hospital data — the comparison that matters for a model meant to be deployed rather than benchmarked. The evidence base remains narrow, covering two outcomes and an external cohort of 100 stays per task from one institution. Code is released under Apache-2.0 as a source-installable Python package (the README's pip install physiojepa does not resolve — nothing is published to PyPI); pretrained weights are not, so reproducing the model means re-running pretraining from the MIMIC-III waveform data.