Data and R scripts

Research using reverse correlation paradigms has become frequent over the last decade. In this paradigm, participants choose between a large number of image pairs, according to a prompt (e.g., “please choose the face that most resembles a trustworthy face”). The image pairs are generated by imposing random noise on the pixels that make up the image of a base face. The patterns of choices by participants are then used to compute an “average” matrix of noise pixels that, when imposed onto the base face, reflect an underlying shared mental construct of the “face” of the attribute. However, examinations of the reliability and validity of this approach remain rare. In the present paper, we examine the effects of filtering out data from individual participants or trials to increase the signal-to-noise ratio of composite images. We focus on order effects of trials, compliance and reliability effects, as well as the diagnostic quality of image pairs. We present different diagnostic methods to examine these three aspects using data from six reverse correlation studies conducted both in-lab and online with diverse samples (i.e., from Burkina Faso, Germany, and the U.S.) using two different base faces (i.e., Black and White). For each of the six studies, we compare the composite images of the complete sample to composite images that exclude non-complier respondents or non-diagnostic image pairs. Our R scripts are publicly available for easy implementation of our suggestions in related research.