Relevance assessment

This task is designed to test the differences between novices' and experts' in formulating of relevance assessments. We employ the formulated queries obtained from the <em>query formulation</em> task to build a single system ranking of candidate relevant documents. Crowd workers were then provided with medical cases (among 113 topics) and a list of top 10 candidate relevant documents. For each query-list of top 10 results, we obtained from 2 experts and 2 novices the relevance judgement in 3-point scale. More details of this task can be found in our paper in references. Fields of the csv file: <strong>dataset</strong>: <em>CLEF_eHealth</em> or <em>OHSUMED</em> <strong>topic_id</strong>: ID of the topic (from 1 to 50 for CLEF_eHealth, from 50 to 113 for OHSUMED) <strong>answerer_type</strong>: <em>expert</em> or <em>novice</em> <strong>answerer_id</strong>: ID of crowd worker <strong>doc_id</strong>: ID of the candidate document in the dataset <strong>relevance_score</strong>: the relevance rate given by the crowd worker