Relevance assessment

This task is designed to test the differences between novices' and experts' in formulating of relevance assessments. We employ the formulated queries obtained from the query formulation task to build a single system ranking of candidate relevant documents. Crowd workers were then provided with medical cases (among 113 topics) and a list of top 10 candidate relevant documents. For each query-list of top 10 results, we obtained from 2 experts and 2 novices the relevance judgement in 3-point scale. More details of this task can be found in our paper in references. Fields of the csv file: dataset: CLEF_eHealth or OHSUMED topic_id: ID of the topic (from 1 to 50 for CLEF_eHealth, from 50 to 113 for OHSUMED) answerer_type: expert or novice answerer_id: ID of crowd worker doc_id: ID of the candidate document in the dataset relevance_score: the relevance rate given by the crowd worker