Web Data Commons Training and Test Sets for Large-Scale Product Matching - Version 2.0

Many e-shops have started to mark-up product data within their HTML pages using the schema.org vocabulary. The Web Data Commons project regularly extracts such data from the Common Crawl, a large public web crawl. The Web Data Commons Training and Test Sets for Large-Scale Product Matching contain product offers from different e-shops in the form of binary product pairs (with corresponding label “match” or “no match”) for four product categories, computers, cameras, watches and shoes. In order to support the evaluation of machine learning-based matching methods, the data is split into training, validation and test sets. For each product category, we provide training sets in four different sizes (2.000-70.000 pairs). Furthermore there are sets of ids for each training set for a possible validation split (stratified random draw) available. The test set for each product category consists of 1.100 product pairs. The labels of the test sets were manually checked while those of the training sets were derived using shared product identifiers from the Web weak supervision. The data stems from the WDC Product Data Corpus for Large-Scale Product Matching - Version 2.0 which consists of 26 million product offers originating from 79 thousand websites. For more information and download links for the corpus itself, please follow the links below.众多电商平台已开始借助schema.org词汇表（schema.org），在其HTML页面中为产品数据添加标注。Web数据通用库（Web Data Commons）项目会定期从大型公开网页爬虫库Common Crawl（Common Crawl）中提取此类标注数据。面向大规模产品匹配的Web数据通用库训练与测试集（Web Data Commons Training and Test Sets for Large-Scale Product Matching）包含来自不同电商平台的产品报价数据，以二元产品对的形式呈现，并附带“匹配”或“不匹配”的对应标签，涵盖计算机、相机、手表与鞋履四大产品类别。为支持基于机器学习的匹配方法的评估工作，该数据集被划分为训练集、验证集与测试集三部分。针对每一类产品，我们提供四种不同规模的训练集（样本量区间为2000至70000个产品对）。此外，我们还为每一组训练集提供了对应的ID集合，可用于后续的验证集划分（采用分层随机抽样方式）。每一类产品对应的测试集包含1100个产品对。测试集的标签均经过人工核查，而训练集的标签则通过网络共享产品标识符，采用弱监督方式生成。本数据集源自面向大规模产品匹配的WDC产品数据语料库（WDC Product Data Corpus for Large-Scale Product Matching）2.0版本，该语料库包含来自79000家网站的2600万条产品报价数据。如需获取该语料库的更多信息与下载链接，请点击下方链接。