NiCLIP: neuroimaging contrastive language-image pretraining model for predicting text from brain activation images

Predicting cognitive processes from brain activation maps remains a key challenge in neuroscience. Meta-analytic functional decoding provides an estimation of behavioral profiles associated with specific brain regions; however, existing methods fail to capture the full semantic content of the neuroimaging literature. To address this, we present NiCLIP, a contrastive language-image model trained on over 23,000 articles to predict cognitive tasks, concepts, and domains from brain activation patterns.

NiCLIP achieves optimal performance when using full-text articles and a curated cognitive ontology with precise task-concept-domain mappings. Our results show that NiCLIP accurately predicts cognitive tasks from Human Connectome Project group-level maps across multiple domains (e.g., emotion, language, motor) and precisely characterizes the functional roles of specific brain regions, including the amygdala, hippocampus, and temporoparietal junction. However, NiCLIP showed limitations with noisy subject-level activation maps.

NiCLIP represents a significant advancement in functional decoding, offering a powerful tool for hypothesis generation and scientific discovery. Special thanks to the FIU Instructional & Research Computing Center (IRCC, http://ircc.fiu.edu) for providing the HPC and computing resources that contributed to the research results reported in this paper. Funding for this project was provided by NIH R01-MH096906.

Department of Physics, Florida International University, Miami, FL, USA Department of Psychology, University of Texas at Austin, Austin, TX, USA Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK Department of Neurology and Neurosurgery, McGill University, Montreal, QC, Canada The authors declare no competing interests. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material.

You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Peraza, J.A., Kent, J.D., Nichols, T.E. et al. NiCLIP: neuroimaging contrastive language-image pretraining model for predicting text from brain activation images.