Enter any query shorter than an old tweet (140 characters), or click on "I'm feeling brainy" to try it out!
Gia H. Ngo, Minh Nguyen, Nancy F. Chen, Mert R. Sabuncu. Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2021
Over the years, neuroscientists have reported thousands of imaging results from functional magnetic resonance imaging (fMRI) and described their findings in scientific articles. Using 13,000 published neuroimaging articles, we trained an artificial neural network to predict brain activation maps from snippets of text. The figure below shows the schematic of the model and the training procedure.
1Text2Brain consists of a text encoder based on SciBert language model, and a 3-D convolutional neural network (CNN). 2For each training sample, a text snippet is randomly picked from the title, keywords, sentences from abstract or discussion of a published research article. For each article, a 3-D volumetric brain image created by placing a 9-mm full-width-half-max (FWHM) Gaussian sphere at an activation coordinate reported in the article. This procedure results in several text-image sample pairs per research article. 3Given a text query input, the text encoder produces an encoding vector, which is then reshaped and passed through the 3-D CNN to generate a brain image. During training, the model is optimized using back propagation from mean squared error (MSE) between the predicted and target brain images.
Huge thanks to all the neuroscienntists whose papers we have used and the authors of Neurosynth and Neuroquery for making their data public.
Prediction from our Text2Brain model can be retrieved programmatically via the /api endpoint via a POST request.
The endpoint expects a JSON of the format { "query": "your-query-string" }. Note that for query longer than 140 characters, the query will be truncated to 140 characters before being passed as input to Text2Brain.
A successful response is a JSON with the following fields:
query: actual query passed to the model (might be different from the input query due to truncation)download_path: download path for the predicted brain volume in nii.gz formatrelated_articles: array of 5 articles (in our training dataset) with the average activation map that best match the predicted brain image, measured by Pearson's correlation. Each entry array is a JSON with the following fields:pmid: PubMed ID of the article
author: authors of the article
title: Title of the article
correlation: per-voxel Pearson's correlation coefficient betwen the predicted brain image and the article's average activation map
Example:
Please let us know if you spot something strange in the predicted brain maps. We would love to improve the model!