PhysioOmni

PhysioOmni is a foundation model for multimodal physiological signals, covering electroencephalography (EEG), electrocardiography (ECG), electrooculography (EOG), and electromyography (EMG) in one pretrained backbone. Physiological recording setups differ enormously between studies and between the lab and the clinic: a polysomnography rig captures EEG, EOG, and chin EMG, a wearable may capture ECG alone, and electrodes fail mid-session. Models built for a fixed modality set degrade sharply or become unusable when that set changes, which is the practical obstacle PhysioOmni addresses — it is designed so that any subset of the four modalities can be supplied at inference without retraining.

The model comes from Nanyang Technological University and Shanghai Jiao Tong University, released as a preprint in April 2025 with code and weights published later that year. Its authors previously built LaBraM, and PhysioOmni extends that model's vector-quantized tokenizer plus masked-prediction recipe from EEG alone to four signal types recorded together, alongside contemporaneous EEG-only backbones such as CBraMod.

The central design choice is a decoupled tokenizer. Instead of one shared representation per sample, PhysioOmni splits every modality's features into a modality-invariant part and a modality-specific part, quantized against separate codebooks. Shared structure survives when a channel drops out; modality-specific structure is reconstructed from what remains.

#Key Features

• Decoupled multimodal tokenizer: Five learnable codebooks — one shared across all modalities and one private per modality, each 8192 entries of dimension 64 — separate modality-invariant from modality-specific structure during vector quantization.
• Missing-modality robustness by construction: Cross-modal reconstruction and a disentangling objective during pretraining let the encoders accept any subset of EEG, ECG, EOG, and EMG at test time, including combinations never seen in that configuration downstream.
• Prototype-alignment fine-tuning: Downstream adaptation pulls each sample toward a learnable class prototype shared across modalities, so unimodal and multimodal inputs land in the same decision space rather than requiring separate heads.
• Modality-aware preprocessing: Each signal type keeps its own band-pass, notch, and resampling settings and its own patch length — 1 second for EEG, 0.5 second for EOG, 0.2 second for ECG and EMG — instead of forcing a common sampling rate.
• Released code and weights: Training scripts for all three stages are public, and the pretrained tokenizer and backbone checkpoints are distributed on HuggingFace under CC-BY-4.0. The code repository itself carries no license file, so only the weights come with explicit terms of reuse.

#Technical Details

Training proceeds in three stages: vector-quantized tokenizer training, masked signal modeling, and resilient fine-tuning with prototype alignment. The encoders combine a small temporal convolution stem with a 12-layer transformer, 10 attention heads, and a hidden size of 200 for EEG and 100 for the other modalities. Masking ratios are 0.5 for EEG and EMG and 0.7 for EOG and ECG. Pretraining draws on datasets that mostly carry three or more simultaneous modalities, including the Temple University Hospital EEG Corpus, DEAP, Sleep-EDF, the CAP Sleep Database, the GX transcranial-stimulation dataset, and an in-house 54-recording collection; training ran on four A100-80G GPUs. Evaluation spans four downstream tasks: seven-class emotion recognition on SEED-VII, five-stage sleep classification on HMC, full-body motion regression on FBM, and binary workload detection on EEGMAT. On SEED-VII with all three available modalities, PhysioOmni reaches 0.3642 balanced accuracy against 0.3475 for a feature-fusion baseline and 0.3456 for LaBraM-Base on EEG; restricted to EEG alone at test time it still scores 0.3479, above every EEG-only baseline. Ablations show that removing prototype alignment, cross-modal reconstruction, the disentangling loss, the shared codebook, or pretraining itself each degrades performance.

#Applications

PhysioOmni suits groups building classifiers over clinical or wearable physiological recordings where label budgets are small and sensor availability is inconsistent — sleep scoring from polysomnography, affective-state decoding from combined EEG/EOG/ECG, movement decoding for prosthetics and rehabilitation, and mental workload monitoring. Its robustness to missing inputs matters most in deployment rather than benchmarking: a model fine-tuned on a full research montage can still run on a reduced consumer headset or on a session where one electrode failed, avoiding the separate per-configuration models such settings usually require.

#Impact

PhysioOmni pushes biosignal foundation models past the single-modality framing that has dominated the area, showing that a shared/private codebook split yields representations that transfer across four signal types and survive arbitrary modality dropout. Limitations are worth stating plainly: the preprint has not been peer reviewed, absolute accuracies on hard tasks like seven-class emotion recognition remain low in absolute terms even where they lead the field, the pretraining corpus includes an unreleased private dataset, and the HuggingFace repository ships weights with no accompanying model card, leaving the GitHub README and the paper as the only documentation.