Loading…
Spatiotemporal vision transformer that turns a resting-state fMRI scan into 4D brain network maps, supervised by windowed ICA components.
No providers recorded yet. Browse all providers
Independent component analysis returns one static spatial map per brain network plus a separate time course, and that factorization carries an assumption the biology does not honor: that a network's spatial footprint is fixed and only its amplitude changes from second to second. Sliding-window and phase-synchrony methods recover part of the missing motion, but still operate on spatially fixed nodes, so a network that changes shape or location mid-scan cannot say so.
st-DenseViT reframes the problem as dense prediction, the computer-vision task of labeling every element of the input. It predicts an activation at every voxel at every time point, yielding a 4D map per network that evolves across the scan — a soft parcellation in which voxels contribute to several networks with weights that change over time. It was developed by Behnam Kazemivash and Vince Calhoun's group at the Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), a joint center of Georgia State University, Georgia Tech, and Emory University, with collaborators at the University of Maryland School of Medicine, the Laureate Institute for Brain Research, and Albert Einstein College of Medicine.
No ground truth exists for what a network looks like at a given instant, so such maps cannot simply be learned by supervision. Training instead uses spatially constrained windowed ICA components as weak supervision — noisy targets the model reproduces and, in practice, learns to clean up. The trained checkpoint then maps a new subject's fMRI to dynamic network maps with no ICA step at inference, the same fixed-checkpoint generative setup as the group's CondAttn-DDPM, which synthesizes static 3D networks by diffusion.
The backbone patchifies each 3D fMRI volume, applies a linear projection, and adds a learnable positional embedding. The decoder head is a layer norm and fully connected layer, a fixed sine-cosine positional encoding of time indices, three depth-wise 3D transposed-convolution layers with kernel sizes 7, 5, and 9 — the last dilated by 2 — and a point-wise Conv3D with randomized leaky ReLU. The loss divides a log-cosh regression term by the structural similarity between prediction and prior. Supervision comes from windowed spatially constrained ICA, window 30 and stride 1, against NeuroMark components derived from more than 800 healthy-control scans.
Training used 508 fMRI datasets pooled from MPRC, FBIRN, and COBRE — 315 controls and 193 subjects with schizophrenia — subsampled to 10 time points each. The configuration is deliberately small: embedding dimension 96, 6 attention heads, depth 1, patch size 5, batch size 2 on two A40 GPUs, Adam at learning rate 0.01 with weight decay 0.1 for up to 150 epochs with early stopping. On held-out subjects the cerebellar network scores best under the space-time encoder at mean absolute relative error 0.10 and homogeneity 0.84, the default mode network reaches the highest structural similarity at 0.87, and the salience network is consistently hardest. Ablations show patch size dominates quality — default mode similarity falls from 0.87 at patch size 5 to 0.51 at 7 and 0.37 at 9 — while cutting the input from 10 time points to 3 flattens the output into near-stationary maps.
The intended use is individualized dynamic network profiling: one subject's scan goes into a trained checkpoint and out comes how each of their functional networks moves during the session, rather than an atlas-average footprint. That suits psychiatric neuroimaging, where diagnosis-linked differences are increasingly thought to live in transient network configurations, and offers methods researchers a nonlinear alternative to windowed ICA.
The work establishes dense prediction as a viable framing for brain dynamics: weak supervision from an imperfect linear decomposition can teach a transformer something the decomposition itself cannot express. Its limits: the training corpus is modest, evaluation is a held-out split of the same pooled cohorts rather than an independent one, and the network vocabulary is bounded by the NeuroMark template used as supervision, so the model refines a fixed set of priors rather than discovering networks. The implementation is released under an MIT license as part of the authors' Scepter framework, but no trained weights and no model card accompany it.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.