ORCAS-I

ORCAS-I is an annotated version of ORCAS dataset (Craswell et al., 2020) annotated with user intents using weak supervision. It allows you to train your algorithm on various types of user intents. Those intents are initially taken from Broder's (2002) classification: informational, navigational and transactional. We also refined this classification and added two subcategories inside the informational category: factual and instrumental. If the intent did not get any label inside the informational category it was classified as abstain. ORCAS-I consists of the following files:a) ORCAS-I-18M.tsvA complete ORCAS data set which contains 18 million unique query-urls pairs.dataset size: 18,823,602unique queries: 10,405,339unique URLs: 1,422,029unique domains: 241,199 b) ORCAS-I-2M.tsvA 2M subset of ORCAS-I-18M.tsv that we used for our experiments with different machine learning algorithms.dataset size: 2,000,000unique queries: 1,796,652unique URLs: 618,679unique domains: 126,001Both ORCAS-I-18M and ORCAS-I-2M contain the following columns:qid: the id of the queryquery: the text of the queryurl: the url that the user clickeddid: the document from TREC deep learning track that the url leads tolevel_1: first level of annotation which has three top level categories: informational, navigational and transactionallevel_2: second level of annotation (only classifies according to factual and instrumental categories, so all the other intents in the column are classified as abstain)label: final intent label. Provides the annotation for informational, navigational and transactional categories and also for factual, instrumental and abstain subcategoriesdata_split: either 'train' or 'validation' that corresponds to split used during the original experimentsYou can train your classifier either on the 3 top level categories (column 'level_1') or on the full taxonomy (column 'label'). c) ORCAS-I-gold.tsvThis is a test file that contains 1000 randomly selected queries from the full dataset (they are excluded from the 2M sample). These queries were manually annotated by two IR specialists. dataset size: 1,000unique queries: 1,000unique URLs: 995unique domains: 700ORCAS-I-gold contains the following columns:qid: the id of the queryquery: the text of the queryurl: the url that the user clickeddid: the document from TREC deep learning track that the url leads tolabel_manual - manually annotated intentdata_split: always equal to 'test'

ORCAS-I是ORCAS数据集（Craswell等人，2020）的标注版本，采用弱监督（weak supervision）方式标注用户意图。本数据集支持针对多种用户意图训练算法。这些意图最初源自Broder（2002）提出的分类体系：信息型、导航型与交易型。我们对该分类体系进行了细化，在信息型类别下新增两个子类别：事实型与工具型。若某意图未被归入信息型类别下的任意子类，则将其标记为弃权（abstain）。

ORCAS-I包含以下文件：
a) ORCAS-I-18M.tsv
完整的ORCAS数据集，包含1800万条唯一查询-URL对。
数据集规模：18,823,602
唯一查询数：10,405,339
唯一URL数：1,422,029
唯一域名数：241,199

b) ORCAS-I-2M.tsv
ORCAS-I-18M.tsv的200万条子集，我们在针对不同机器学习算法的实验中使用了该子集。
数据集规模：2,000,000
唯一查询数：1,796,652
唯一URL数：618,679
唯一域名数：126,001

ORCAS-I-18M与ORCAS-I-2M均包含以下字段：
qid：查询ID
query：查询文本
url：用户点击的URL
did：该URL指向的TREC深度学习跟踪任务中的文档
level_1：一级标注，包含三个顶级类别：信息型、导航型与交易型
level_2：二级标注（仅针对事实型与工具型类别进行分类，其余所有意图在该列中均标记为弃权）
label：最终意图标签，涵盖信息型、导航型、交易型三大类别，以及事实型、工具型与弃权三大子类别
data_split：取值为「train（训练集）」与「validation（验证集）」，对应原始实验中采用的数据集划分方式

你既可以基于3个顶级类别（「level_1」列）训练分类器，也可以基于完整分类体系（「label」列）开展训练。

c) ORCAS-I-gold.tsv
该测试文件包含从完整数据集中随机选取的1000条查询（这些查询未被纳入200万子集）。这些查询由两名信息检索（Information Retrieval，IR）专家进行了手动标注。
数据集规模：1,000
唯一查询数：1,000
唯一URL数：995
唯一域名数：700

ORCAS-I-gold包含以下字段：
qid：查询ID
query：查询文本
url：用户点击的URL
did：该URL指向的TREC深度学习跟踪任务中的文档
label_manual：手动标注的意图
data_split：固定为「test（测试集）」

搜集汇总

背景与挑战

背景概述

ORCAS-I是一个基于ORCAS数据集的用户意图标注数据集，使用弱监督方法将用户意图分为信息型、导航型和交易型三大类，并在信息型下细分为事实型和工具型子类别。数据集包含约1880万条查询-URL对的完整版本、200万条子集用于机器学习实验，以及1000条手动标注的测试集，适用于训练和评估用户意图分类模型，支持多级分类任务。