Estimated confusion matrix comparing gender inferences made by our raw procedure (genderize.io and MTurk) with those made by our calibrated inference strategy, which is based on expert analysis of sampled strata.

This matrix estimates that our raw procedure is correct 97.5% of the time (sum of the diagonal elements). However, we expect the actual number of women to be approximately 1% lower than the raw estimates (8.9% vs 9.9%), primarily due to the overestimate of women by the genderize.io API. Throughout the remainder of our study, we use the calibrated gender inferences.