BENDR

An EEG classifier trained on one motor-imagery dataset rarely survives a change of amplifier, electrode cap, or subject. The usual response is to keep the network shallow and refit it for every recording context, so each study starts from scratch even though decades of unlabeled clinical EEG already sit in hospital archives. BENDR, from the SPOClab group at the University of Toronto with the Vector Institute and the Li Ka Shing Knowledge Institute, asks whether that archive can be used the way a text corpus is used for a language model.

The mechanism it borrows is wav2vec 2.0, the contrastive self-supervised recipe for raw speech. A stack of strided 1D convolutions compresses raw EEG into a sequence of vectors — the BErt-inspired Neural Data Representations the model is named for — and a transformer reads that sequence with contiguous spans masked out. The training signal is not reconstruction, which on a smooth, highly autocorrelated signal degenerates into learning interpolation, but discrimination: at each masked position the transformer output must be closer in cosine similarity to the true encoded vector than to distractors drawn from elsewhere in the same recording. Nothing about the objective is task-specific, so a single run can consume clinical recordings that carry no BCI labels at all.

Published in Frontiers in Human Neuroscience in 2021, BENDR is among the earliest transformer models pretrained on EEG at corpus scale, and its released checkpoints made it a reference point that later EEG foundation models — including CodeBrain and the Large Cognition Model — measure themselves against.

#Key Features

• Contrastive masked-span objective: contiguous runs of ten encoded vectors are masked with probability 0.065 per position, and the model must identify the true representation among twenty distractors sampled from the same sequence at a temperature of 0.1.
• A hardware-agnostic input contract: every dataset is resampled to 256 Hz and mapped onto the 19 unambiguous 10/20 electrodes plus one relative-amplitude channel, so recordings with 2, 22, or 64 electrodes all enter as 20 channels.
• Released pretrained checkpoints: encoder and contextualizer weights are published as release assets, and the repository's downstream script loads them directly.
• Sequence-length independence: position comes from a grouped convolution rather than fixed positional embeddings, so the same weights accept a two-second BCI trial or a thirty-second sleep epoch.
• Transfer to disjoint datasets: pretraining runs once on clinical recordings, and evaluation happens on five public datasets with different hardware, subjects, and tasks.

#Technical Details

The convolutional stage is six blocks of 1D convolution, group normalization, and GELU, each 512 filters wide, downsampling by a factor of 96 — one 512-dimensional vector per 375 ms, an effective rate of about 2.67 Hz. The contextualizer is an 8-layer, 8-head transformer with a model dimension of 1536 and a feed-forward width of 3076, using T-Fixup initialization with the internal layer norms removed, LayerDrop 0.01, and dropout 0.15. The two released checkpoints hold roughly 157 million parameters between them: 3.9M in the encoder and 153M in the contextualizer. Pretraining used versions 1.1 and 1.2 of the Temple University Hospital EEG Corpus, about 1.5 TB of European Data Format recordings from over 10,000 people, cut into 60-second windows of 15,360 samples. Across the downstream battery the best configurations reach 86.7 balanced accuracy on the PhysioNet motor movement/imagery set, 42.6% accuracy on the four-class BCI Competition IV 2a, 0.65 AUROC on error-related negativity, 0.72 balanced accuracy on Sleep-EDF staging, and 0.72 AUROC on a P300 speller.

#Applications

The practical use is as a starting point for EEG classification when labels are scarce: motor-imagery and P300 brain-computer interfaces, error-potential detection, and clinical sleep staging all fine-tune from the same weights. Because the input contract normalizes montage and sampling rate, a lab with a 22-channel research amplifier and a group working from two-electrode polysomnography load the same checkpoint. The architecture has since been reimplemented in the braindecode library, which serves the pretrained weights as a feature extractor.

#Impact

BENDR showed that one self-supervised EEG model can produce representations whose contrastive accuracy is nearly uniform across unseen subjects and hardware — around 0.90 on four of the five evaluation datasets. The authors are candid that most fine-tuned results are not state of the art: the strongest configuration on four of five tasks bypassed the transformer entirely, average-pooling the representations into a linear classifier while continuing to train the convolutional stage, and configurations that froze the encoder outright often stayed only marginally above chance. The pretraining corpus is access-controlled, requiring registration with Temple University, and the repository carries no license file and is pinned to a 2021-era version of the authors' DN3 library. What endures is the recipe: mask spans of a learned EEG representation, discriminate rather than reconstruct, normalize the montage on the way in.