Loading…
Generative foundation model for intracranial EEG whose attention separates content, time and channel, so one checkpoint fits any electrode montage.
No providers recorded yet. Browse all providers
Every patient implanted with intracranial electrodes gets a montage designed around their own clinical question: a different number of contacts, in different structures, at depths chosen by the surgical team. There is no channel 7 that means the same thing in two patients, so a transformer with an absolute positional embedding per channel learns something that does not survive the move to the next subject. Intracranial EEG (iEEG) models have accordingly been refit per patient, and generalized poorly when they were not.
MVPFormer answers this with multi-variate parallel attention (MVPA), a self-attention variant that splits each query-key interaction into three additive terms: a content term computed from the two segment embeddings with no positional information at all, a time term depending only on relative distance in time, and a channel term depending only on relative distance in space. Because both positional terms are relative, and the channel term is learned rather than supplied, the model builds its own implicit map of how electrodes relate instead of being told where they sit; a montage it has never seen is simply a different set of relative offsets. It was built at IBM Research with the Sleep-Wake-Epilepsy-Center of Inselspital, Bern University Hospital, and published at ICLR 2026.
MVPFormer is trained on intracranial recordings, not scalp EEG, which separates it from most electrophysiology foundation models. Its closest neighbours are the intracranial models it is benchmarked against — Brant, Brant-2 and BrainBERT — all of which use vanilla attention and, in this evaluation, fail to carry a seizure detector onto an unfamiliar cohort.
The backbone is a Llama 2-style decoder with Megatron-LM-style parallel attention and MLP blocks, fed by a db4 wavelet encoder that maps each five-second, single-channel segment to a learned embedding. Two sizes exist: MVPFormer-S at roughly 74M parameters (12 layers, 12 heads, model dimension 768) and MVPFormer-M at 1.2B (24 layers, 16 heads, model dimension 2048), both using grouped-query attention. Content attention is restricted to a local window of 10 segments while the time and channel terms span the full context, keeping cost subquadratic in context length; with a Triton kernel called FlashMVPA, effective context exceeds 10,000 tokens on one A100-80GB. Pretraining ran 1.2M steps on eight such GPUs over two weeks.
On seizure detection across 50 unseen SWEC subjects, MVPFormer reaches a mean Cohen's kappa of 0.61 against board-certified annotations — above the 0.53 threshold the paper takes as expert-level inter-rater agreement — with an F1 of 0.59 and 0.15 false positives per hour. The vanilla-attention baselines score 0.11 (MV-Llama), 0.06 (Brant-2) and 0.00 (BrainBERT) on the same protocol, and on the external MAYO and FNUSA cohorts MVPFormer reaches F1 0.36 and 0.46. On the four Brain TreeBank decoding tasks, MVPFormer-S reaches 0.83 accuracy on pitch, 0.88 on volume, 0.87 on onset and 0.90 on speech.
The immediate use is clinical: long-term epilepsy monitoring produces days of multi-channel recording a neurologist must read for ictal events, and a detector holding expert-level agreement at 0.15 false positives per hour is usable as a first pass over that volume. The pretrained backbone also gives groups with small implanted cohorts a starting point for decoding work — the Brain TreeBank tasks were reached by LoRA fine-tuning, not training from scratch — and the SWEC corpus is independently useful for benchmarking.
Public iEEG corpora have been small — tens of hours, a handful of subjects — and the large ones private, holding back foundation models in a domain where signal is abundant but access is not. Releasing 9,328 hours with ictal labels, together with code and weights, changes what an outside group can attempt. Code and checkpoints are tagged Apache-2.0, but the repository carries its own disclaimer that the software may be used for research only and in particular must not be used for diagnostic purposes; the two have not been reconciled. Two limits are worth noting: SWEC omits electrode locations for privacy, so models that exploit anatomical position cannot be trained on it, and PopT — which does use electrode coordinates — still leads on the onset and speech tasks. The 18 pretraining subjects also supply roughly 400M tokens, short of the compute-optimal budget for even the smaller model.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.