ORCAS-I

ORCAS-I is an annotated version of ORCAS dataset (Craswell et al., 2020) annotated with user intents using weak supervision. It allows you to train your algorithm on various types of user intents. Those intents are initially taken from Broder's (2002) classification: informational, navigational and transactional. We also refined this classification and added two subcategories inside the informational category: factual and instrumental. If the intent did not get any label inside the informational category it was classified as abstain. ORCAS-I consists of the following files:a) ORCAS-I-18M.tsvA complete ORCAS data set which contains 18 million unique query-urls pairs.dataset size: 18,823,602unique queries: 10,405,339unique URLs: 1,422,029unique domains: 241,199 b) ORCAS-I-2M.tsvA 2M subset of ORCAS-I-18M.tsv that we used for our experiments with different machine learning algorithms.dataset size: 2,000,000unique queries: 1,796,652unique URLs: 618,679unique domains: 126,001Both ORCAS-I-18M and ORCAS-I-2M contain the following columns:qid: the id of the queryquery: the text of the queryurl: the url that the user clickeddid: the document from TREC deep learning track that the url leads tolevel_1: first level of annotation which has three top level categories: informational, navigational and transactionallevel_2: second level of annotation (only classifies according to factual and instrumental categories, so all the other intents in the column are classified as abstain)label: final intent label. Provides the annotation for informational, navigational and transactional categories and also for factual, instrumental and abstain subcategoriesdata_split: either 'train' or 'validation' that corresponds to split used during the original experimentsYou can train your classifier either on the 3 top level categories (column 'level_1') or on the full taxonomy (column 'label'). c) ORCAS-I-gold.tsvThis is a test file that contains 1000 randomly selected queries from the full dataset (they are excluded from the 2M sample). These queries were manually annotated by two IR specialists. dataset size: 1,000unique queries: 1,000unique URLs: 995unique domains: 700ORCAS-I-gold contains the following columns:qid: the id of the queryquery: the text of the queryurl: the url that the user clickeddid: the document from TREC deep learning track that the url leads tolabel_manual - manually annotated intentdata_split: always equal to 'test'