Crowdsourcing Document Similarity Judgements

This is the data obtained from crowdsourcing tasks which ask workers to provide similarity metrics between pairs of documents. Each document, as well as each pair, has a unique ID. We provide crowd workers with the pairs through three different task variations: Variation 1: We showed workers 5 pairs of documents and, for each, asked them to rate their similarity in a 4-level Likert scale (None, Low, Medium, High), tell us a confidence level of how sure they were (from 0 to 4) and a written reason as to why they chose that similarity level. For quality reasons, two of the 5 pairs were golden-standards, which means we knew their ratings already and checked the workers' responses. They had to give the golden pair with the higher similarity a higher score than the other golden pair, otherwise, their answer would be rejected. Variation 2: We repeated variation 1 but with a slight alteration: instead of a Likert scale for the similarity score, we asked for a Magnitude Estimation, which is any number above 0. It could be 1, 0.0001, 1000, 42, as long as it was coherent, as in a more similar pair had a higher score than a less similar pair and vice-versa; Variation 3: We showed workers 5 rankings. Each ranking had a main document and 3 auxiliary documents to be compared against the main one. They also had to report a confidence score and give a short written reason, just like variation 1. The first ranking is a golden-standard, and we knew the values for the 3 pairs in it (the pairs were the main document paired with each of the 3 auxiliary documents), and they had to give the golden pair with the highest similarity a higher rank than the one with the lower similarity. The raw results from the tasks are recorded in the JSON file CrowdResults.json. For a description of its contents, please read the file CrowdResults_README.md. These raw annotations from the crowd were then parsed into the three CSVs you see, each corresponding to the aggregated results from one of the task variations. <em>final_scores_likert.csv</em> is the resulting scores for each pair using the variation 1 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>similarity_crowd_simple_maj </em>stores the simple majority result from the crowd's annotations; <em>similarity_crowd_simple_mean </em>stores the mean of the crowd's annotations; <em>similarity_crowd_simple_median </em>stores the median of the crowd's annotations; <em>final_scores_magnitude.csv</em> is the resulting scores for each pair using the variation 2 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>scaled_similarity_worker</em> is the magnitude score scaled based on worker's behaviours <em>scaled_similarity_worker_docset </em>is the magnitude score scaled based both on the worker's behaviour and on the pair <em>final_scores_ranking.csv</em> is the resulting scores for each pair using the variation 3 tasks; <em>pair_id </em>is a unique identifier for each pair; <em>similarity_alg </em>is the similarity assigned to the pair of documents from an automated similarity algorithm; <em>relation </em>is the type of relationship shown by the pair, where smaller values indicate more similar pairs; <em>mean_similarity</em> is the mean ranking from that value This dataset was built and used as part of the TheyBuyForYou project.