CardioState-JEPA

An ECG electrode, a fingertip pulse sensor, and a chest microphone all record the same heartbeat, but none sees it at the same instant. Electrical activation comes first; the first heart sound follows roughly 75 ms later as the valves close; the peripheral pulse arrives another 125–160 ms after that. Align those streams by timestamp, as ordinary multimodal training does, and the model is asked to match an electrical deflection to a completely different phase of the same beat — so it learns surface appearance instead of physiology.

CardioState-JEPA treats that delay as something to estimate rather than ignore. Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) are modelled as three renderings of one hidden cardiac state, each seen through its own transduction and its own point along the cardiovascular path. Modality-specific stems map each waveform into a common token space, a single shared Transformer encoder produces a common cardiac code, and training predicts masked latent states rather than reconstructing waveforms. A learned delay aligner shifts the cross-modal target to the corresponding cardiac time before comparing.

It was developed by Hamza Shafiq, Aaqib Saeed and colleagues at Eindhoven University of Technology with Singapore Management University, and released as a preprint in August 2026. It departs from the single-sensor lineage of ECG-FM, ECGFounder, HeartLang and the same group's D-BETA on ECG, and PaPaGei and Pulse-PPG on PPG, by making the correspondence between sensors supervision rather than a nuisance.

#Key Features

• One encoder across three sensing modalities: ECG, PPG, and PCG update the same parameters, so a multi-sensor device needs one representation model, not one per signal.
• Learned physiological delay: a bounded delay head predicts the per-token offset and gathers the target with a Gaussian kernel centred at the shifted time, regressed onto R-peak-to-first-heart-sound and pulse arrival anchors wherever clean beats are detected.
• Latent prediction, not reconstruction: the objective compares predicted and momentum-encoder codes, so nothing rewards fitting sensor noise or fine morphology.
• A curriculum matched to data scarcity: Stage I learns within-modality structure from abundant unimodal recordings; Stage II spends the scarce synchronized recordings on alignment while still sampling unimodal data so per-modality codes do not drift.
• Frozen-encoder transfer: results come from freezing the encoder and training only a linear head, on corpora disjoint from pretraining, with patient-disjoint splits.

#Technical Details

The shared encoder follows a ViT-B configuration — 12 layers, 12 attention heads, hidden dimension 768 — fed by per-modality stems that pair a strided convolution with a multi-scale depthwise block sized to short events such as the QRS complex and the PPG upstroke, all emitting 16–25 tokens/s. A momentum encoder supplies stop-gradient targets, and the objective adds cross-modal VICReg, delay supervision, and a cardiac-phase term to the Smooth-L1 prediction losses. Unimodal pretraining uses MIMIC-IV-ECG (roughly 800K 12-lead recordings at 500 Hz), the MIMIC-III-Ext-PPG corpus at 125 Hz, and BMD-HS for PCG at 4000 Hz; paired supervision comes from VitalDB, EPHNOGRAM, and SensSmartTech. Stage I runs 300K steps and Stage II a further 200K, on a single NVIDIA H100.

Across 25 downstream tasks the frozen encoder reaches a mean macro-AUROC of 84.1 over 18 ECG settings (six datasets by three label fractions) against 68.5 for the strongest self-supervised ECG-only baseline, lifts mean PPG classification macro-AUROC from 72.2 to 80.4 while cutting mean regression MAE from 10.9 to 9.1, and reaches 97.9 macro-AUROC on CirCor murmur detection. The modality silhouette of the pooled codes falls from 0.121 after Stage I to −0.006 after Stage II, so the shared space becomes sensor-invariant while class structure sharpens.

#Applications

The frozen encoder is a general feature extractor for cardiac waveforms in clinical and wearable settings alike: diagnostic ECG classification and atrial fibrillation detection, murmur and abnormal-heart-sound screening from auscultation, and the wearable regression tasks of heart rate, respiratory rate, blood pressure, SpO2, and heart-rate variability, plus stress and activity recognition. Because a linear head suffices, groups with small labelled cohorts can probe it without the annotation budget supervised cardiac models assume.

#Impact

CardioState-JEPA makes the case that heterogeneous cardiac sensors can mutually supervise one model, and that the offsets between them are learnable structure rather than an obstacle. The shared encoder checkpoint is released under a non-commercial licence, behind a Hugging Face gate that requires a logged-in account accepting the terms; the code is public but carries no licence at all. The work remains a preprint awaiting peer review, and its limitations are worth weighing: the paired and trimodal corpora are far smaller than the unimodal ones, PCG pretraining data is the scarcest of the three, and on ECG the trimodal model sits within noise of the best bimodal ECG+PPG variant, so the third modality's clearest benefit is for PPG and PCG. The delay aligner assumes a detectable reference beat, full fine-tuning and larger encoders are left to future work, and the headline gains are measured against self-supervised signal-only baselines rather than the text- and label-supervised models reported alongside them.