CEReBrO

EEG is multi-channel and non-stationary, and a foundation model for it has to balance two different structures at once: how a signal evolves within one electrode over time, and how electrodes relate to each other at a given instant. Existing models tended to sacrifice one of them. Some tokenize each channel independently and never model spatial correlation; others aggregate all channels into a single token and lose temporal granularity; the ones that do both, such as LaBraM, pay for it with a separately pretrained neural tokenizer or with hundreds of millions of parameters — a problem for the hardware EEG actually runs on.

CEReBrO, from Alexandru Dimofte, Glenn Anta Bucagu, and colleagues at the University of Zurich and the Integrated Systems Laboratory at ETH Zurich, is a deliberately small EEG foundation model built around an alternating attention mechanism. Tokens are patches of a single channel, so temporal detail survives tokenization, and successive encoder layers alternate between attending across channels at a fixed time index and attending across time within a fixed channel. Because neither layer ever forms the full quadratic attention matrix over channel-time pairs, memory and runtime drop sharply while both structures still get modelled inside one encoder.

Released in January 2025 in three sizes, it is the direct predecessor of S-CEReBrO, the same group's streaming variant, and a sibling of their later EEG models FEMBA and LUNA.

#Key Features

• Alternating attention: Odd layers attend over channels at each time step (O(C²·Np)), even layers attend over patches within each channel (O(C·Np²)), against O((C·Np)²) for standard self-attention — measured at over 6x less GPU memory and over 2x faster forward passes at long sequence lengths.
• Per-channel patch tokenization: Waveforms are cut into non-overlapping patches per channel and given both positional and channel embeddings, so intra- and inter-channel correlations are modelled jointly.
• Variable montage handling: A shared learnable padding token, with attention scores forced to zero, lets a single encoder ingest recordings from 18 to 36 channels without interpolating the electrode layout.
• Single-stage masked pretraining: Masked autoencoding directly on raw waveforms, with no discrete codebook or tokenizer to pretrain first, and a reconstruction loss that adds visible patches at weight 0.1 to stop the encoder collapsing to identity. Pretraining and every downstream benchmark use openly available corpora.

#Technical Details

CEReBrO is an encoder-only transformer released at 3.58M, 39.95M, and 85.15M parameters (8, 10, and 12 layers; embedding dimensions 192, 576, and 768; 12 heads throughout; patch size 64). Pretraining is masked autoencoding at a 0.5 mask ratio on the Temple University EEG Corpus — over 20,000 hours from more than 10,000 subjects across 18-36 channel configurations — with a global batch of 4,096, AdamW, and a peak learning rate of 1.25e-3. Downstream, the encoder feeds a single linear layer and can be frozen or fully fine-tuned; global mean pooling replaces the usual CLS token.

On SEED emotion classification the 85M model reaches 68.21% accuracy and 0.6845 F1, well ahead of LaBraM at 46M (57.93%, 0.5899), and even the 3.58M model beats every supervised baseline. On TUAB abnormality classification it reaches 81.67% balanced accuracy and 0.8916 AUROC, matching LaBraM at up to 4x fewer parameters. On the Neonate seizure dataset it reaches 0.875 AUROC and 0.690 AUPR — the strongest result among general-purpose EEG foundation models, though the paper's own table places it behind the task-specific STATENET at 0.910, a nuance the abstract's "new benchmarks in seizure detection" phrasing elides. On MoBI gait regression the large model exceeds LaBraM Base and Large on R². Ablations attribute a +1.3% balanced accuracy, +0.0146 AUPR, and +0.0125 AUROC gain on TUAB to alternating attention over standard self-attention.

#Applications

The target deployments are clinical and consumer EEG on constrained hardware: neonatal and epilepsy seizure detection, routine abnormality screening of clinical recordings, affective state estimation, and movement decoding from mobile brain-body imaging. Because one pretrained encoder transfers to all four with only a linear head, a group with a few hundred labelled recordings can build a classifier without assembling a pretraining corpus of its own.

#Impact

CEReBrO makes the case that architectural fit matters more than scale for EEG: a factorized attention pattern matched to the channel-time structure of the signal recovers most of what much larger models achieve, at sizes a smartphone or an edge TPU can run. The alternating attention idea carried directly into the group's later work, where S-CEReBrO extends it with fixed-size windows to bound memory on continuous recordings. Pretrained CEReBrO checkpoints have not been published — the lab's Hugging Face organization hosts its later EEG models but not this one — and the paper remains a preprint. The authors note that LaBraM frequently surpasses CEReBrO at matched size, and that all baseline numbers are taken from the original publications rather than retrained.