Gold standard of ISO 25010 quality (sub)characteristics in online user reviews

In this gold standard, we annotated a stratfied random sample of 360 app store reviews according to all eight characteristics and 31 subcharacteristics of the software product quality taxonomy in ISO/IEC 25010:2011. The reviews were split into 1,358 individual sentences. Four taggers annotated each sentence using the ISO 25010 standard as tagging schema. The sample contained apps from six categories (entertainment, productivity, social media, messaging, games, smart products), with two apps per category (one paid, one free), with user reviews for each app taken from three app stores (Apple, Google, and Amazon), for a total of 36 sources. Of each source, we sampled 10 reviews (two per rating from 1 star to 5 stars). More details are provided in Section 2a of our paper. The gold standard can be used, among other things, for machine learning and as a catalog of examples of utterances in online user feedback regarding particular quality (sub-)characteristics.