OK, Computer, what are these books about? - data files
收藏资源简介：
The core of this experiment is the use of the entity-fishing algorithm, as created and deployed by DARIAH. In the most simple terms: it scans texts for terms that can be linked to Wikipedia pages. Based on the algorithm, new keywords are added to the book descriptions, plus a list of relevant Wikipedia pages.For this experiment, the full text of 4125 books and chapters – available in the OAPEN Library – is scanned, resulting in a data file of over 25 million entries. In other words, on average the algorithm found roughly 6,100 ‘hits’ for each publication. When only the most common terms per publication are selected, does this result in a useful description of its content?The data file OK_Computer_results contains a list of open access books and chapters descriptions found in the OAPEN Library, combined with Wikipedia entries found using the entity-fishing algorithm, plus several actions to filter out only the terms which describe the publication best. Each book or chapter is available in the OAPEN Library (www.oapen.org), see the column HANDLE/The data file nerd_oapen_response_database contains the complete data set. The other text files contain R code to manipulate the file nerd_oapen_response_database.Description of nerd_oapen_response_database:The data is divided into the following columns:Data DescriptionOAPEN_ID Unique ID of the publication in the OAPEN LibraryrawName The entity as it appears in the textnerd_score Disambiguation confidence scorenerd_selection_score Selection confidence score, indicates how certain the disambiguated entity is actually valid for the text mentionwikipediaExternalRef ID of the Wikipedia pagewiki_URL URL of the Wikipedia pagetype NER class of the entitydomains Description of subject domainEach book may contain more than one occurrence of the same entity. The nerd_score and the nerd_selection_score may vary. This allows researchers to count the number of occurrences and use this as an additional method to assess the contents of the book. The OAPEN_ID refers to the identifier of the title in the OAPEN Library.For more information about the entity-fishing query processing service see https://nerd.readthedocs.io/en/latest/restAPI.html#response.
本实验的核心是使用由DARIAH开发并部署的entity-fishing（实体链接算法）。简言之，该算法会扫描文本，检索可关联至维基百科页面的术语。基于该算法，可为图书描述新增关键词以及相关维基百科页面清单。 本次实验共扫描OAPEN文库中收录的4125种图书及章节的全文，最终生成包含超2500万条条目的数据文件。换言之，该算法平均可为每篇出版物检索到约6100个命中项。若仅选取每篇出版物中最具代表性的术语，能否得到对其内容兼具实用性的描述？ OK_Computer_results数据文件收录了OAPEN文库中开放获取图书及章节的描述列表，结合了通过entity-fishing算法识别出的维基百科条目，同时包含若干筛选操作，仅保留最能体现出版物内容的术语。每种图书或章节均可在OAPEN文库（www.oapen.org）中获取，对应列名为HANDLE。 nerd_oapen_response_database数据文件则包含完整数据集。其余文本文件均为用于处理nerd_oapen_response_database的R代码。 nerd_oapen_response_database数据集说明： 该数据集包含以下列： OAPEN_ID：OAPEN文库中出版物的唯一标识符 rawName：文本中出现的实体原称 nerd_score：消歧置信度得分 nerd_selection_score：筛选置信度得分，用于表明经消歧后的实体对当前文本提及内容的适配可信度 wikipediaExternalRef：维基百科页面的外部引用ID wiki_URL：维基百科页面的URL type：实体的命名实体识别（Named Entity Recognition，简称NER）类别 domains：主题领域描述 同一实体可能在单本图书或章节中多次出现，此时nerd_score与nerd_selection_score可能存在差异。研究人员可通过统计实体出现次数，以此作为评估出版物内容的额外方法。此处的OAPEN_ID指向OAPEN文库中对应作品的标识符。 若需了解更多关于entity-fishing查询处理服务的详情，请访问https://nerd.readthedocs.io/en/latest/restAPI.html#response.
