Webis Gmane Email Corpus 2019

The Webis Gmane Email Corpus 2019 is a dataset of more than 153 million parsed and segmented emails crawled between February and May 2019 from gmane.io covering more than 20 years of public mailing lists. The dataset has been published as a resource at ACL 2020. The dataset comes as a set of Gzip-compressed files containing line-based JSON in the Elasticsearch bulk format. Each data record consists of two lines: <pre><code class="language-json">{"index": {"_id": "&lt;urn:uuid:c1d95e4b-0f43-46c7-a99e-c575d1d8e1ce&gt;"}} {"headers": {"header name": "header value", ...}, "text_plain": "plaintext body", "lang": "en", "segments": [{"end": 99, "label": "paragraph", "begin": 0}, ...], "group": "gmane group name"} </code></pre> The first line is the Elasticsearch index action with a document UUID, the second one the actual parsed email with a (reduced and anonymized) set of headers, the detected language, the original Gmane group name and the predicted content segments as character spans. The Gzip files are splittable every 1,000 records (line pairs) for parallel processing in, e.g., Hadoop. Available email headers are: message_id date (yyyy-MM-dd HH:mm:ssZZ) subject from to cc in_reply_to references list_id Available segment classes are: paragraph closing inline_headers log_data mua_signature patch personal_signature quotation quotation_marker raw_code salutation section_heading tabular technical visual_separator Find more information about the dataset and the segmentation model at webis.de. If you are using this resource in your work, please cite it as: <pre><code>@InProceedings{stein:2020o, author = {Janek Bevendorff and Khalid Al-Khatib and Martin Potthast and Benno Stein}, booktitle = {58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)}, month = jul, publisher = {Association for Computational Linguistics}, site = {Seattle, USA}, title = {{Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis}}, year = 2020 } </code></pre>Webis Gmane邮件语料库2019（Webis Gmane Email Corpus 2019）是一个包含超1.53亿条经解析与分段处理的邮件的数据集，该数据集于2019年2月至5月间从gmane.io爬取，涵盖了超过20年的公开邮件列表内容。该数据集已于2020年国际计算语言学协会年会（ACL 2020）上作为研究资源发布。 该数据集以Gzip压缩文件集的形式提供，文件内采用基于行的JSON格式，符合Elasticsearch批量导入格式。每条数据记录包含两行，格式示例如下： json {"index": {"_id": "<urn:uuid:c1d95e4b-0f43-46c7-a99e-c575d1d8e1ce>"}} {"headers": {"header name": "header value", ...}, "text_plain": "plaintext body", "lang": "en", "segments": [{"end": 99, "label": "paragraph", "begin": 0}, ...], "group": "gmane group name"} 第一行为带有文档UUID（Universally Unique Identifier）的Elasticsearch索引操作行，第二行为实际解析后的邮件内容，包含经精简与匿名化处理的邮件头集合、检测得到的语言类型、原始Gmane邮件组名称，以及以字符跨度形式标注的预测内容分段。 该Gzip压缩文件支持每1000条记录（即每行对）进行拆分，可用于并行处理场景，例如Hadoop环境。 可用的邮件头字段包括：message_id、date（格式为yyyy-MM-dd HH:mm:ssZZ）、subject、from、to、cc、in_reply_to、references、list_id。 可用的分段类别包括：paragraph（段落）、closing（收尾语）、inline_headers（内嵌邮件头）、log_data（日志数据）、mua_signature（邮件用户代理签名）、patch（补丁）、personal_signature（个人签名）、quotation（引用内容）、quotation_marker（引用标记）、raw_code（原始代码）、salutation（称呼）、section_heading（章节标题）、tabular（表格数据）、technical（技术内容）、visual_separator（视觉分隔符）。 如需了解该数据集及分段模型的更多信息，请访问webis.de。 若您在研究工作中使用该资源，请按以下格式引用： bibtex @InProceedings{stein:2020o, author = {Janek Bevendorff and Khalid Al-Khatib and Martin Potthast and Benno Stein}, booktitle = {58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)}, month = jul, publisher = {Association for Computational Linguistics}, site = {Seattle, USA}, title = {{Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis}}, year = 2020 }