NiCLIP

NiCLIP is a contrastive language-image pretraining model for neuroimaging functional decoding, developed by Julio Peraza and Angela Laird at Florida International University with collaborators at the University of Texas at Austin, the University of Oxford, and McGill University, and posted as a preprint in June 2025. It answers a question neuroscientists ask constantly and answer poorly: given a map of where the brain activated, which cognitive processes were engaged. This reverse-inference problem is normally handled by meta-analytic decoders such as Neurosynth and GC-LDA, which match activation patterns to terms counted in the literature — representations that discard semantic context.

NiCLIP replaces term counting with language-model semantics. Text from more than 23,000 full-text fMRI articles is embedded with a large language model, activation coordinates reported in those articles are converted into brain maps, and a CLIP-style contrastive objective pulls matched text-brain pairs together in a shared latent space. Predictions are then made through a cognitive ontology rather than a free vocabulary: the Cognitive Atlas supplies named tasks with written definitions, and Bayes' theorem converts latent-space similarity into a posterior probability per task, propagated to concepts and domains.

The model builds on the NeuroConText contrastive framework and extends it in two directions: ontology-guided probabilistic decoding, and neuroscience-specialized language models such as BrainGPT for text encoding. Where NeuroVLM generates in both directions between literature text and brain maps, NiCLIP is built for structured decoding. Code is released under Apache-2.0 as the braindec Python package, with pretrained models and embeddings hosted on OSF.

#Key Features

• Ontology-guided probabilistic decoding: Predictions are drawn from the Cognitive Atlas ontology rather than arbitrary terms, and task posteriors propagate through task-to-concept and concept-to-domain mappings, giving three levels of abstraction.
• LLM text encoding over full articles: Text embeddings come from 7B-parameter language models, and training on full article bodies rather than abstracts gives better retrieval and decoding across every configuration tested.
• Coordinate-based literature at scale: Training on reported activation coordinates rather than shared statistical maps taps the largest body of neuroimaging publications, covering tasks sparsely represented in image-sharing repositories such as NeuroVault.
• Region-level functional characterization: Beyond whole-brain maps, the model profiles individual regions, recovering established associations such as the amygdala with emotion, and covering the hippocampus, insula, striatum, and temporoparietal junction.
• Packaged inference: The Python package ships a command-line predictor and a Gradio application returning ranked cognitive tasks for an uploaded activation map.

#Technical Details

Training data are drawn from PubMed Central open-access fMRI articles retrieved with Pubget, giving a cross-validated corpus of 23,865 articles. Reported foci are converted into activation maps by multilevel kernel density analysis and reduced with the DiFuMo 512-region continuous parcellation; text is embedded with a pretrained LLM. Both streams pass through learned encoders — a projection block followed by two residual blocks — into a shared latent space, trained with InfoNCE contrastive loss at batch size 128, learning rate 5e-4, weight decay 0.1, and up to 50 epochs with early stopping.

Four text encoders were compared: BrainGPT-7B-v0.2, its base model Mistral-7B-v0.1, BrainGPT-7B-v0.1, and Llama-2-7b-chat-hf. The best retrieval configuration, BrainGPT-7B-v0.2 on full article bodies, reaches Recall@10 of 33.56% and Recall@100 of 71.95%, with a Mix&Match score of 90.04 — numerically close to its base Mistral model. Functional decoding was evaluated zero-shot on group-level task-fMRI maps from the Human Connectome Project across seven domains: emotion, gambling, language, motor, relational, social, and working memory. Pairing full-text training with a curated reduced Cognitive Atlas ontology and task name plus definition embeddings reaches 62.86% Recall@4 for tasks, 43.57% for concepts, and 90.48% Recall@2 for domains, against a maximum of 20.71% Recall@4 for the baselines.

#Applications

The immediate use is hypothesis generation from an activation map a researcher already has: upload the map, read off the cognitive tasks, concepts, and domains most consistent with the pattern, and frame an interpretation anchored in the literature rather than the analyst's expectations. The same machinery characterizes the functional profile of a region of interest, useful in parcellation work and in designing follow-up experiments. Because decoding runs against a named ontology, outputs feed into meta-analytic workflows and automated annotation of imaging archives.

#Impact

NiCLIP shows that contrastive language-image learning paired with a curated cognitive ontology substantially outperforms the term-frequency decoders that have served neuroimaging meta-analysis for over a decade, gaining more than 40 percentage points in recall over Neurosynth and GC-LDA on HCP domains. It also produces a useful negative result: neuroscience-tuned LLMs perform about as well as their base models here, so the leverage lies in text scope and ontology quality rather than domain-specific fine-tuning. The clearest limitation is stated by the authors: performance degrades on noisy subject-level maps, reaching only 38.19% at the task level, so the model suits group-level maps rather than individual-subject inference. The work remains a preprint under a CC BY-NC-ND license.