A simple statistical misunderstanding is leading many neuroscientists astray in their use of machine learning tools, according to a new paper in the Journal of Neuroscience Methods: Exceeding chance level by chance. As the authors, French neuroscientists Etienne Combrisson and Karim Jerbi, describe the issue:
Machine learning techniques are increasingly used in neuroscience to classify brain signals. Decoding performance is reflected by how much the classification results depart from the rate achieved by purely random classification.
Suppose you record activity from my brain while I am looking at a series of images of people. Some of the people are male, some are female. You want to determine whether there is something about my brain activity (a feature or pattern) that's different between those two classes of stimuli (male and female). Now suppose you find a pattern that allows you to 'read my mind' and determine whether I'm looking at a ...
