Brain2Qwerty

Brain2Qwerty is a non-invasive brain-to-text system built by Meta AI (FAIR) in collaboration with the Basque Center on Cognition, Brain and Language (BCBL). It decodes the sentences a person types on a QWERTY keyboard directly from brain activity captured with magnetoencephalography (MEG) or electroencephalography (EEG), without any surgical implant. Participants type memorized sentences while their brain signals are recorded, and the model learns to reconstruct the character sequence they produced.

The work addresses a central limitation of brain-computer interfaces for communication: the highest-performing speech and text neuroprostheses depend on implanted electrodes, which require surgery and carry attendant medical risk. Brain2Qwerty tests how far entirely non-invasive recordings can be pushed, framing communication decoding as a typing task that ties neural activity to a well-defined motor and linguistic output.

Its central idea is an end-to-end pipeline that pairs a neural signal encoder with a language model, so that the decoder can exploit the statistics of natural language rather than reading out characters in isolation. A first generation established the approach on MEG and EEG; a second generation substantially raises accuracy by adapting a pretrained large language model to each participant's neural data.

#Key Features

• Fully non-invasive decoding: Reconstructs typed text from MEG or EEG recordings with no implanted electrodes, avoiding the surgical risk of intracortical interfaces.
• End-to-end architecture: A convolutional encoder over sensor channels feeds a deep transformer, and a character-level language model corrects and constrains the decoded output.
• Language-model integration: The second generation fine-tunes a pretrained LLM backbone on neural signals using per-subject LoRA adapters that are then averaged, letting the decoder leverage semantic context for more coherent sentences.
• Modality comparison: Trained and evaluated on both MEG and EEG, quantifying how much the richer magnetic signal improves decoding over scalp electrical recordings.
• Open code and data: Training code is released on GitHub under a CC BY-NC 4.0 license, and the first-generation Spanish MEG dataset is publicly available on Hugging Face.

#Technical Details

The first-generation model was trained and evaluated on 35 healthy volunteers who typed memorized sentences during MEG or EEG acquisition. On MEG it reached an average character error rate (CER) of about 32%, with the best participant near 19%, while EEG was markedly harder at roughly 67% CER. The architecture combines a convolutional module that embeds windows of multichannel sensor data, a transformer that models temporal structure, and a character-level language model that decodes and denoises the predicted text.

The second generation focuses on MEG and scales the linguistic component. It uses roughly 10 hours of recordings from each of 9 participants, spanning about 22,000 sentences, and adapts a pretrained language-model backbone (ranging from 0.6B to 4B parameters) to individual neural data through LoRA adapters whose weights are averaged across subjects. This raises decoding to 61% average word accuracy, with the top participant reaching 78%, compared with roughly 8% word accuracy for earlier non-invasive baselines. The first-generation study is peer-reviewed and published in Nature Neuroscience; the second-generation dataset remains under embargo pending publication, and the repository ships training code rather than an off-the-shelf inference checkpoint.

#Applications

Brain2Qwerty is a research system for non-invasive brain-computer interface and communication neuroscience. It offers a benchmark for how accurately typed language can be recovered from MEG and EEG, a testbed for coupling neural decoders with large language models, and a tool for studying the motor and higher-level cognitive processes that underlie sentence production. Because MEG hardware is large, shielded, and non-portable, and because the models require per-subject training, the system is a laboratory research platform rather than a clinical or consumer device, but it establishes methods that assistive-communication research can build on.

#Impact

Brain2Qwerty demonstrates that non-invasive recordings can decode typed sentences at accuracies far above prior scalp-based approaches, narrowing the gap with implanted interfaces without surgery. Its peer-reviewed publication, open training code, and released dataset make the approach reproducible and extensible, and its use of an adapted language-model backbone illustrates a broadly applicable recipe for grounding neural decoders in linguistic priors. The main constraints are practical: MEG's bulk and cost, the need for extensive per-participant data, and the fact that the task is constrained typing rather than free, open-vocabulary speech.