Crowdsourcing Topical Relevance With Crowdtruth

This repository contains the crowdsourcing annotations for topical relevance referenced in the following paper:

Oana Inel, Giannis Haralabopoulos, Dan Li, Christophe Van Gysel, Zoltán Szlávik, Elena Simperl, Evangelos Kanoulas and Lora Aroyo: Studying Topical Relevance with Evidence-based Crowdsourcing. CIKM 2018.

If you find this data useful in your research, please consider citing:

<pre>@inproceedings{inel2018studying,
title={Studying Topical Relevance with Evidence-based Crowdsourcing},
author={Inel, Oana and Haralabopoulos, Giannis and Li, Dan and Van Gysel, Christophe and Szl{\'a}vik, Zolt{\'a}n and Simperl, Elena and Kanoulas, Evangelos and Aroyo, Lora},
booktitle={Proceedings of the 27th ACM International Conference on Information and Knowledge Management},
pages={1253--1262},
year={2018},
organization={ACM}
}</pre>

<strong>Running the notebooks</strong>

To run and regenerate the results, you need to install the stable version of the <em><strong>crowdtruth==2.0</strong></em> package from PyPI using:<br>
pip install crowdtruth==2.0<br>