Loading…
Transformer neural speech decoder for intracranial ECoG and sEEG contacts at arbitrary cortical positions, generalizing to unseen participants.
No providers recorded yet. Browse all providers
Two epilepsy patients can carry identical 8 × 8 electrocorticography grids and still share nothing at grid position (3, 5): in one that contact sits over ventral sensorimotor cortex, in the other over superior temporal gyrus. A decoder that locates electrodes by grid index therefore learns weights whose meaning changes from person to person — which is why intracranial speech decoders have almost always been fit one participant at a time, and why depth electrodes, which follow no grid, were largely unusable.
SwinTW — Swin transformer with temporal windowing — replaces the grid index with anatomy. Each electrode becomes a token positioned by its MNI coordinate in a standardized brain template plus a learned embedding of the cortical region it occupies, and attention between two electrodes is biased by that anatomical relationship rather than by adjacency on a grid. Windowing applies only along the time axis; spatially, attention runs across every electrode at once, so any number of contacts in any arrangement — an 8 × 8 grid, subdural strips, stereo-EEG (sEEG) depth probes, or a mixture — is a valid input. Built at New York University and published in the Journal of Neural Engineering in January 2025, it borrows windowed attention from computer vision's Swin Transformer but is not that model, which appears here only as a baseline.
This entry describes the multi-patient SwinTW. The architecture is also trained per participant, but only the multi-patient regime is fitted across a cohort with no subject-specific layers, so one fixed checkpoint can be applied to someone who contributed nothing to training. That sets it apart from decoders such as Multi-User Speech BCI Decoder, which pool across users through per-user input layers that still need data from each new user.
SwinTW stacks three stages of transformer blocks combining spatial and temporally windowed attention with temporal patch merging, then upsamples with transposed temporal convolution before a prediction head emits speech parameters. Training uses Adam at a learning rate of 5 × 10⁻⁴. The cohort is 52 native English speakers undergoing epilepsy monitoring at NYU — 43 with 8 × 8 ECoG grids at 10 mm spacing over perisylvian cortex, 9 with sEEG only — performing five single-word production tasks over a 50-word set (400 trials each, 350 train / 50 test). High-gamma power (70–150 Hz) is the input.
Decoding is scored as Pearson correlation (PCC) against ground-truth spectrograms. Subject-specific models on grid electrodes reached PCC = 0.817 across 43 participants; on the same 64 contacts SwinTW averaged 0.825 against 0.804 for the group's earlier convolutional ResNet decoder and 0.785 for their 3D Swin transformer. Off-grid strips and depth contacts raised this to 0.838 (N = 39), and sEEG-only models reached 0.798 (N = 9). A multi-patient model trained on 15 participants scored 0.837, on par with 15 individually trained models at 0.831; leave-one-out cross-validation gave a mean PCC of 0.765 on held-out participants.
The work targets speech neuroprostheses for people who cannot speak, and two properties matter clinically. sEEG-only decoding works, and sEEG probes — unlike subdural grids, which need a large craniotomy — are routinely implanted in chronic neurosurgery. A cross-subject checkpoint also removes the need for paired acoustic-neural recordings from a patient who can no longer speak, though the synthesizer stage is still fitted to that participant from their speech audio alone.
SwinTW is, by the authors' account, the first speech decoder trained across multiple intracranial participants with no subject-specific layers, and it shows that anatomical coordinates are a workable substitute for the grid indices earlier architectures depended on — an idea that carries over to other neural decoding targets. Related intracranial models such as Brant learn general-purpose sEEG representations; SwinTW is specialized for speech acoustics. Its limits are stated plainly: accuracy on unseen participants is lower and less consistent than subject-specific fitting, the multi-patient experiments used only the 8 × 8 grid data, and evaluation is spectrogram correlation on isolated word production rather than continuous conversational speech. No implementation or trained checkpoint has been released; one example ECoG recording is public, with the remaining data available from the corresponding author.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.