Loading…
Multimodal neuroimaging model fusing resting-state fMRI and EEG, pretrained without labels across spatial, temporal, and spectral domains.
No providers recorded yet. Browse all providers
fMRI and EEG watch the same brain with opposite blind spots. fMRI localizes activity to millimeters but samples the BOLD signal only every couple of seconds; EEG follows electrical activity at millisecond resolution but smears it across the scalp. The standard way to combine them is to concatenate the two feature vectors before a classifier, which treats them as independent votes and discards what makes them complementary — that in any one view of the data, one modality is systematically the better teacher.
MCSP (Multi-modal Cross-domain Self-supervised Pre-training) replaces concatenation with directed transfer. Built by Yu Zhang's group at Lehigh University with collaborators at Children's National Hospital and the University of Texas at Austin, it encodes each modality in three views — a spatial connectome, a temporal series, and its frequency transform — then distills knowledge along the axis where it is richer. In the spatial domain fMRI teaches EEG; in the temporal and frequency domains EEG teaches fMRI. A contrastive objective runs alongside, pulling a subject's three views together and pushing different subjects apart, so pretraining needs no diagnostic labels.
It sits alongside single-modality brain models such as BrainMass, NeuroSTORM and the later BrainGFM from the same lab, all of which learn from fMRI alone. A single encoder set, pretrained once on an unlabeled union of cohorts, is then fitted with a task-agnostic MLP head for each downstream diagnosis.
The full model holds 3.4M parameters — 1.2M spatial, 1.1M temporal, 1.1M frequency. Both modalities are parcellated to the Schaefer 100-ROI atlas: fMRI time series are resampled to length 200, while EEG is downsampled to 25,000 points and cut into 125 segments of 200 to align with them. EEG connectivity comes from source-localized power envelopes, restricted to the alpha band (8–12 Hz). Pretraining pools five cohorts — ADHD-200 (830 subjects), ABIDE I (1,102), ABIDE II (990), EMBARC (324 fMRI, 308 EEG) and the Healthy Brain Network (2,282 fMRI, 1,594 eye-open and 1,744 eye-closed EEG) — into three unlabeled corpora for fMRI, EEG, and paired scans. Training used Adam at a learning rate of 0.0005 for 50 epochs on a single RTX 4090, batch 128 for pretraining and 32 for fine-tuning, over 10 runs of 10-fold cross-validation. On ADHD-200 the pretrained model reaches 79.6% AUROC against 68.1% for Dynamic BNT, with 70.2% on ABIDE I and 71.5% on ABIDE II. On the multimodal cohorts, pretraining lifts EMBARC depression grading from 66.3% to 69.5% AUROC and sex classification from 68.5% to 72.6%, and HBN diagnosis from 74.3% to 77.6% for MDD and 68.2% to 71.6% for ASD.
The model targets groups holding resting-state fMRI, resting-state EEG, or both for psychiatric and neurodevelopmental cohorts. Because the pretraining objective needs no labels, a lab can pretrain on its full archive and fine-tune on the small labeled subset it actually has — the setting that dominates clinical neuroimaging, where scans outnumber diagnoses. The cross-scenario experiments cover the transfers such labs face: fMRI to EEG, ABIDE I to ABIDE II, one ADHD-200 site to another. Region-importance rankings additionally surface candidate biomarkers, highlighting the lingual, superior parietal, superior frontal and inferior temporal cortices for depression severity.
MCSP was published in Neural Networks after appearing as a preprint, and it treats fMRI–EEG fusion as an asymmetric distillation problem rather than a feature concatenation problem. Two limits frame its results. The five downstream cohorts sit inside the unlabeled pretraining union, so the demonstrated generalization is across tasks, sites and modalities rather than to a held-out external cohort. And the scope is narrow by construction — resting-state fMRI only, alpha-band EEG only, a single 100-region parcellation — so other frequency bands, task paradigms or atlases mean retraining. No code or pretrained weights accompany the paper, leaving the architecture and losses reproducible from the text but the checkpoints not directly reusable.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.