DIVER-0

EEG foundation models face a structural problem that image and text models do not: no two labs record the same way. Electrode counts, montages, and channel orderings differ between datasets, so a model that implicitly memorizes "channel 3 is Fz" fails the moment it meets a new recording setup. Most existing EEG transformers also factorize attention, processing the spatial (across-electrode) and temporal (across-time) axes in separate stages — a design choice that CBraMod made explicit and that keeps the model from directly representing interactions that are simultaneously spatial and temporal.

DIVER-0, from Seoul National University with collaborators at Brookhaven National Laboratory, argues that both choices should be revisited. It applies full attention over the flattened channel-by-time patch grid, and it supplies the positional information that segregated attention would otherwise provide through two targeted mechanisms: Rotary Position Embedding along the temporal axis, and a binary attention bias that distinguishes same-channel from cross-channel interactions without naming any individual electrode. That binary formulation is what buys channel-permutation equivariance — permuting the input channels permutes the output identically, so nothing in the model depends on channel order.

The paper's second contribution, Sliding Temporal Conditional Positional Encoding (STCPE), preserves both temporal translation equivariance and channel permutation equivariance at once, which prior conditional positional encodings for EEG do not. Presented as a spotlight at the ICML 2025 Workshop on Generative AI and Biology, DIVER-0 is framed as an architectural study establishing design principles rather than a finished released model.

#Key Features

• Full spatio-temporal attention: A single attention operation spans the entire channel-by-time patch grid, rather than alternating between separate spatial and temporal stages as in prior EEG transformers.
• Channel-permutation equivariance: Binary attention biases differentiate same-channel from cross-channel pairs without encoding electrode identity, so the model adapts to arbitrary montages unseen during pretraining and gives consistent results under channel permutation at both pretraining and fine-tuning.
• Sliding Temporal Conditional Positional Encoding: A transformer block slides across the temporal axis with a 7-second window, aggregating patches from all channels at each step to compute positional information while retaining both equivariance properties.
• Dual-pathway patch embedding: Each patch is encoded by a patch-wise CNN for temporal features and an FFT-based spectral pathway, combined by element-wise addition.
• Data-efficient pretraining: Competitive downstream results come from pretraining on just 10% of the Temple University Hospital EEG Corpus.

#Technical Details

The backbone stacks 12 DIVER encoder blocks with a hidden dimension of 200, feed-forward dimension of 800, and 10 attention heads, pretrained by masked patch reconstruction. Pretraining uses the Temple University Hospital EEG Corpus — 69,652 clinical recordings from 14,987 subjects, roughly 30,000 hours — following CBraMod's preprocessing: 0.3-75 Hz bandpass, 60 Hz notch filter, resampling to 200 Hz, 30-second non-overlapping segments, amplitude rejection above 100 microvolts, and 19 standard 10-20 channels. Training ran on four A100 40GB GPUs with AdamW.

Fine-tuning covers two downstream tasks with different electrode configurations. On FACED nine-class emotion recognition (32 channels, 123 subjects), DIVER-0 pretrained on 10% of TUEG reaches 59.2% balanced accuracy and 54.0% Cohen's kappa, ahead of CBraMod at 55.1% and LaBraM-Base at 52.7%. On PhysioNet-MI four-class motor imagery (64 channels), it reaches 62.8% balanced accuracy against CBraMod's 64.2% and LaBraM-Base's 61.7%. The authors read that split as evidence that unified attention helps for distributed brain processes such as emotion while more localized attention suffices for sensorimotor tasks. Ablations confirm each component contributes on FACED, and a permutation study shows performance holds across intact and permuted channel orderings in both pretraining and fine-tuning.

#Applications

DIVER-0 is aimed at settings where the recording montage cannot be standardized: pooling EEG across research groups, transferring a pretrained backbone between clinical sites with different caps, or deploying consumer and clinical devices with reduced or non-standard electrode sets. For groups building brain-computer interfaces, permutation equivariance removes a class of silent failure in which a model degrades because channels arrive in an unexpected order. The architectural findings — that binary channel biases plus RoPE can replace segregated attention, and that conditional positional encoding can be made doubly equivariant — apply to any multi-channel biosignal with variable sensor layouts.

#Impact

DIVER-0 contributes a concrete formalization of channel-permutation equivariance for EEG foundation models and shows it can be achieved without giving up accuracy, which matters for a field whose datasets are irreducibly heterogeneous. Its scope is deliberately limited: this is an 11-page workshop paper evaluated on two downstream tasks, pretrained on a tenth of the available corpus, with no released code or weights, and the authors themselves flag broader benchmarking and full-corpus scaling as future work. The tradeoff they identify is also real — binary channel differentiation buys equivariance at the cost of finer-grained electrode-specific discrimination, which shows up as a deficit on the spatially localized motor-imagery task.