Estimated error rates for faculty rank transitions from 2011 to 2017.

Estimated error rates (expressed as percentages; rows sum to 100%) for all possible transitions of the form X → Y, where X is the rank of a faculty member in the 2011 manual census (where “New” indicates that they were unobserved in 2011) and Y is their rank in 2017 (where “Gone” indicates that they were unobserved at any institution in 2017). To construct this confusion matrix, we used crowdsourcing to determine Y for a 10% uniform random sample of the 2011 and 2017 faculty, and compared those titles (columns) to the output of our crawler (rows).