ECG-SSL Long-Context Encoder

Atrial fibrillation is often paroxysmal: it starts, runs for minutes or hours, and stops. A cardiologist reading a Holter tape reaches a diagnosis by watching how the rhythm behaves over that span, not by inspecting one beat. Most self-supervised ECG models are nonetheless trained on windows of five to sixteen seconds, and several recent ones first compress the waveform into a discrete token vocabulary. Both choices buy throughput, and both discard exactly the evidence — slow rhythm variability, baseline drift, individual morphology — that accrues over minutes.

This encoder, developed at the University of Minnesota Twin Cities and accepted at EMBC 2026, tests that trade-off directly and releases the model that comes out of it. A single Transformer backbone is pretrained on ten-minute windows of single-lead ambulatory ECG using a patient-contrastive InfoNCE objective: two windows drawn from different recording segments of the same person are pulled together, windows from other people are pushed apart. No rhythm labels are involved, so the pretraining set is simply raw signal.

The accompanying study varies two axes — input horizon (16 s, 1 min, 5 min, 10 min) and front-end representation (continuous CNN patch embeddings versus fixed vector-quantized tokens) — while holding backbone, optimizer, and data splits constant. The paper credits the 5- and 10-minute models jointly with the strongest performance, but only the 16-second and 10-minute arms ship checkpoints, so the continuous 10-minute encoder is the configuration actually released for reuse; the VQ and randomly initialized arms are deliberately weaker controls. Sibling ECG foundation models such as ECG-FM, ECGFounder and the tokenized HeartLang sit on the short-window or discretized side of exactly the design choices measured here.

#Key Features

• Ten-minute context window: 150,000 samples at 250 Hz are cut into non-overlapping 160-sample (0.64 s) patches, giving the Transformer a 937-token sequence spanning many rhythm episodes rather than a single snapshot.
• Patient-contrastive pretraining: InfoNCE positives are two windows of the same patient from different recorded segments, often days apart and under different activity states, so the objective rewards representations that survive session change.
• Continuous over discretized front end: at matched context and backbone, CNN patch embeddings reach 0.877 frozen AUPRC against 0.610 for a fixed K=256 vector-quantized codebook, evidence that quantization erases clinically informative waveform detail.
• Usable frozen: patient retrieval is pure embedding plus cosine similarity with no training step, and rhythm classification works from a linear probe on the frozen encoder.
• Cross-session patient identity: Recall@1 of 0.907 and Recall@5 of 0.945 on 550 held-out patients never seen during pretraining.

#Technical Details

The backbone is a 6-layer, 8-head Transformer with hidden dimension 256, feedforward expansion factor 4, GELU activations, dropout 0.1 and learned positional embeddings; global mean pooling feeds a two-layer projection head (256 → 256 → 128) producing an L2-normalized embedding. Pretraining uses InfoNCE at temperature 0.1 with 32 patients per step (64 windows, 62 negatives), AdamW at learning rate 3×10⁻⁴ with weight decay 0.05, a cosine schedule with 200 warmup steps, and gradient clipping at norm 1.0.

Training data is Icentia11k, an ambulatory single-lead corpus of 11,000 patients with roughly 50 segments of about 70 minutes each, bandpass filtered 0.5–40 Hz and z-scored per window. Splits are strictly patient-level, reserving 8,800 patients exclusively for self-supervised pretraining. On AFib/AFL versus normal-rhythm classification the frozen 10-minute encoder reaches AUC 0.980 and AUPRC 0.877, rising to 0.989 and 0.960 when fine-tuned end to end. The 16-second encoder, evaluated on the same 10-minute windows by averaging sub-window probabilities, recovers less: Recall@1 falls to 0.737. Fine-tuning trades retrieval for classification, dropping Recall@1 from 0.907 to 0.851.

#Applications

The frozen encoder turns hours of wearable or Holter recording into fixed-length vectors, which makes similarity search practical over archives far too large to read manually: cohort stratification, subgroup discovery, and finding patients whose rhythm resembles an index case. The linear-probe results matter for label-scarce settings, where a few hundred annotated windows are enough to build a usable AFib/AFL screen. The same patient-consistency that enables retrieval is also a privacy consideration — embeddings of de-identified single-lead ECG carry enough individual signature to link recordings back to the same person.

#Impact

The contribution is a measured answer to a design question the ECG foundation model field has mostly answered by assumption, and the checkpoints make the answer reusable rather than merely reported. The source code is released under the MIT license; the checkpoints ship as git-LFS objects with no stated terms of their own, no tagged release, and no model card. The release is also materially incomplete: the data/ package the README documents — datasets, preprocessing, augmentation, splits, and collate functions — is excluded by the repository's own .gitignore and absent from the published tree, so the training and evaluation scripts that import it do not run as shipped and the held-out 550-patient split cannot be rebuilt. Icentia11k is itself distributed under CC BY-NC-SA 4.0, a non-commercial share-alike license that constrains what anyone can do downstream with encoders trained on it. The evidence base is narrow: one dataset, one lead, one downstream task, and no external-corpus transfer, so the generalization claim extends to held-out patients within Icentia11k and no further. A six-layer encoder is also far smaller than ECG foundation models trained on millions of recordings; its value is as a controlled demonstration that context length and a continuous front end matter more than the field's default settings imply.