ggfox00000/dia-earning21-all

[](LICENSE.md) # Earnings 21 The Earnings 21 dataset ( also referred to as earnings21 ) is a 39-hour corpus of earnings calls containing entity dense speech from nine different financial sectors. This corpus is intended to benchmark automatic speech recognition (ASR) systems in the wild with special attention towards named entity recognition (NER). This work has been recently accepted to Interspeech 2021! # Table of Contents * [File Format Overview](#file-format-overview) + [nlp Files](#nlp-files) - [Example](#example-nlp-file) + [wer_tag JSON](#wer_tag-json) - [Example](#example-wer_tag-json) * [Entity Labels](#entity-labels) * [Results and Eval-10](#results) + [Eval-10: A Representative Earnings-21 Subset](#eval-10-a-representative-earnings-21-subset) * [WER Calculation](#wer-calculation) * [Cite this Dataset](#cite-this-dataset) # File Format Overview In the following section, we provide an overview of the file formats we provide with this dataset. ## nlp Files NLP files are `.csv` inspired, pipe-separated text files that contain token and metadata information of a transcript. Each line of a file represents a single transcript token and the metadata associated with it. |Column Title|Description |--|--| | Column 1: `token` | A single token in the transcript. These are typically single words or multiple words with hyphens in between. | | Column 2: `speaker` | A unique integer that associates this token to a specific speaker in an audio | Column 3: `ts` | A float representing the start time of the token, in seconds | Column 4: `endTs` | A float representing the end time of the token, in seconds | Column 5: `punctuation` | A punctuation character that is included at the end of a token that is used when reconstructing the transcript. Example punctuation: `",", ";", ".", "!"`. | Column 6: `case` | A two letter code to denominate the which of four possible casings for this token: <ul><li>`UC` - Denotes a token that has the first character in uppercase and every other character lowercase.</li><li>`LC` - Denotes a token that has every character in lowercase.</li><li>`CA` - Denotes a token that has every character in uppercase.</li><li>`MC` - Denotes a token that doesn’t follow the previous rules. This is the case when upper- and lowercase characters are mixed throughout the token</li></ul> | Column 7: `tags` | Displays one of the several entity tags that are listed in wer_tags in long form - such that the displayed entity here is in the form `ID:ENTITY_CLASS`. | Column 8: `wer_tags` | A list of entity tags that are associated with this token. In this field, only entity IDs should be present. The specific ENTITY_CLASS for each ID can be extracted from an accompanying wer_tags sidecar json. | _**Note that each entity ID is unique to that specific entity. Entities can be comprised of single and multiple tokens. Within a file there can be several entities of the same ENTITY_CLASS but only one entity can be labeled with any given ID.**_ ### Example nlp File `example.nlp` ``` token|speaker|ts|endTs|punctuation|case|tags|wer_tags Good|0||||UC|[]|[] morning|0||||LC|['5:TIME']|['5'] and|0||||LC|[]|[] welcome|0||||LC|[]|[] to|0||||LC|[]|[] the|0||||LC|['6:DATE']|['6'] first|0||||LC|['6:DATE']|['6'] quarter|0||||LC|['6:DATE']|['6'] 2020|0||||CA|['0:YEAR']|['0', '1', '6'] NexGEn|0||||MC|['7:ORG']|['7'] ``` ## wer_tag JSON The wer_tags sidecar JSON is used in combination with an nlp file and exclusively when that file is using the wer_tags column. It is used to provide entity information about each entity ID. It is formatted such that the JSON acts as a list of objects that map the ID of an entity to an object specifying the entity_type as the entity label. The object is formatted such that: ```json "ID":{ "entity_type" : "LABEL" } ``` ### Example wer_tag JSON `example.wer_tags.json` ```json { "0":{ "entity_type" : "YEAR" }, "1":{ "entity_type" : "CARDINAL" }, "5":{ "entity_type" : "TIME" }, "6":{ "entity_type" : "DATE" }, "7":{ "entity_type" : "ORG" } } ``` # Entity Labels In the following table, we provide a list of all possible entity tags we provide in the dataset including a description of each tag and a few examples. | Tags (Entity Classes) | Description | Examples | | --------------------- | ----------- | -------- | | PERSON |Names of people, including fictional people | Hagrid, Jason Chicola, W. E. B. Du Bois | | NORP | Nationalities or religious or political groups | American, Chinese, Republican, Grand Old Party, Roman Catholic | | FAC | Buildings, airports, highways, bridges, etc. | Golden Gate Bridge, the Empire State Building | | ORG | Companies, agencies, institutions, etc. | Rev, General Motors, SEC, NAACP | | GPE | Countries, cities, states, etc. Geopolitical entities. | Italy, US, Boston, New Zealand | | LOC | Non-GPE locations, mountain ranges, bodies of water,etc. | the North, the Rocky Mountains | | PRODUCT | Objects, vehicles, foods, etc. (not services) | Camry, Sufentanil, ARX-02 | | EVENT | Named hurricanes, battles, wars, sports events, etc. | COVID-19, the Spanish Flu, Hurricane Katrina, World War II | | WORK_OF_ART | Titles of books, songs, etc. | Frankenstein, The Mona Lisa | | LAW | Named documents made into laws, etc. | Fox’s Act 1792, Article 5 of the Constitution | | LANGUAGE | Any named language, etc. | Esperanto, Spanish, Swahili | | DATE | Absolute or relative dates or periods, etc. | Q1, the end of last year | | TIME | Times smaller than a day, etc. | Morning, 30 minutes | | PERCENT | Percentage, including "%"', etc. | Approximately 10%, 5% | | MONEY | Monetary values, including unit, etc. | $40 billion, 20 thousand pounds | | QUANTITY | Measurements, as of weight or distance, etc. | 10 kilometers, approximately 80 tons | | ORDINAL | "first", "second", etc. | Third, eighth, 4th | | CARDINAL | Numerals that do not fall under another type, etc. | 20, fifty, 1420 | | ABBREVIATION | An all-caps shortened form of a word or phrase where each letter represents a word. Specifically all Initialisms or Acronyms. | AFK (away from keyboard), RSU (restricted stock unit), FEMA | | WEBSITE | A written out website | www.rev.com, indeed.com | | YEAR | A 4 digit number representing a year | 2020, 2021 | | CONTRACTION | A word or group of words resulting from shortening an original form OR can be transformed into a common form. | I’ll (I will), going to (gonna) | | ALPHANUMERIC | A token that is comprised of letters and numbers | 8th, Q4 | | FALLBACK | A word that does not conform to a normal word. This is usually words that contain an unknown symbol (like &) or words that were only partially spoken (like th-) | Q&M, lo- | # Results Tables found in the paper along with all entity class WER can be found within the `transcripts` directory. ## Eval-10: A Representative Earnings-21 Subset Along with the results found in the paper, we've included a subset denoted as Eval-10 which is a representative 10 hour sample of the full Earnings-21 corpus. This subset is not meant to replace the full dataset but rather allow for researchers to quickly evaluate their systems before running results on the full dataset. WER calculations for all systems on this subset can be found within the same table found in the `transcripts` directory. # WER Calculation All of our analysis on this dataset is done through the use of our newly released [fstalign](https://github.com/revdotcom/fstalign/tree/master) tool. We strongly recommend the use of this tool to quickly get started using the *Earnings-21* dataset. # Cite this Dataset This dataset has been submitted to Interspeech 2021. The paper describing our methods and results can be found on arXiv at https://arxiv.org/pdf/2104.11348.pdf ``` @misc{delrio2021earnings21, title={Earnings-21: A Practical Benchmark for ASR in the Wild}, author={Miguel Del Rio and Natalie Delworth and Ryan Westerman and Michelle Huang and Nishchal Bhandari and Joseph Palakapilly and Quinten McNamara and Joshua Dong and Piotr Zelasko and Miguel Jette}, year={2021}, eprint={2104.11348}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```[](LICENSE.md) # 收益电话会议数据集21（Earnings 21） 收益电话会议数据集21（Earnings 21，亦可简称earnings21）是一段总长39小时的收益电话会议语料库，包含来自9个不同金融行业的实体密集型语音内容。该语料库旨在为真实场景下的自动语音识别（Automatic Speech Recognition, ASR）系统提供基准测试支持，并特别针对命名实体识别（Named Entity Recognition, NER）任务进行了优化设计。本数据集相关研究已被Interspeech 2021会议收录！ # 目录 * [文件格式概览](#file-format-overview) + [nlp文件](#nlp-files) - [示例](#example-nlp-file) + [wer_tag JSON文件](#wer_tag-json) - [示例](#example-wer_tag-json) * [实体标签](#entity-labels) * [实验结果与Eval-10](#results) + [Eval-10：代表性Earnings21子集](#eval-10-a-representative-earnings-21-subset) * [词错误率计算](#wer-calculation) * [数据集引用](#cite-this-dataset) # 文件格式概览 在以下章节中，我们将介绍本数据集配套提供的各类文件格式。 ## nlp文件 nlp文件为受CSV格式启发的竖线分隔文本文件，用于存储转录文本的Token（Token）及元数据信息。文件内每一行对应一个转录Token及其关联的元数据。 |列标题|描述 |--|--| | 第1列：`token` | 转录文本中的单个Token，通常为单个单词或带连字符的多单词组合。 | | 第2列：`speaker` | 用于将该Token关联至音频中特定说话者的唯一整数标识符。 | | 第3列：`ts` | 表示该Token起始时间的浮点数值，单位为秒。 | | 第4列：`endTs` | 表示该Token结束时间的浮点数值，单位为秒。 | | 第5列：`punctuation` | 用于转录文本重构的Token末尾标点符号，支持`",", ";", ".", "!"`等常见标点。 | | 第6列：`case` | 用于标识该Token四种大小写格式的双字母代码：<ul><li>`UC`：表示首字符大写、其余字符小写的Token。</li><li>`LC`：表示所有字符均为小写的Token。</li><li>`CA`：表示所有字符均为大写的Token。</li><li>`MC`：表示不符合上述规则的混合大小写Token，即Token中同时包含大写与小写字符。</li></ul> | | 第7列：`tags` | 以长格式显示实体标签，格式为`ID:ENTITY_CLASS`，即此处展示的实体为`ID:实体类别`。 | | 第8列：`wer_tags` | 与该Token关联的实体标签列表，该字段仅包含实体ID，具体实体类别可从配套的wer_tag JSON附属文件中获取。 | _**请注意：每个实体ID仅对应唯一实体。实体可由单个或多个Token组成。在单个文件中，可存在多个属于同一实体类别的实体，但任意给定ID仅可用于标注一个实体。**_ ### 示例nlp文件 `example.nlp` token|speaker|ts|endTs|punctuation|case|tags|wer_tags Good|0||||UC|[]|[] morning|0||||LC|['5:TIME']|['5'] and|0||||LC|[]|[] welcome|0||||LC|[]|[] to|0||||LC|[]|[] the|0||||LC|['6:DATE']|['6'] first|0||||LC|['6:DATE']|['6'] quarter|0||||LC|['6:DATE']|['6'] 2020|0||||CA|['0:YEAR']|['0', '1', '6'] NexGEn|0||||MC|['7:ORG']|['7'] ## wer_tag JSON文件 wer_tag JSON附属文件需与nlp文件配合使用，且仅在nlp文件使用`wer_tags`列时生效，用于提供每个实体ID对应的实体信息。其格式为以实体ID为键、以包含`entity_type`字段的对象为值的JSON结构，其中`entity_type`字段即为实体类别标签，格式如下： json "ID":{ "entity_type" : "LABEL" } ### 示例wer_tag JSON文件 `example.wer_tags.json` json { "0":{ "entity_type" : "YEAR" }, "1":{ "entity_type" : "CARDINAL" }, "5":{ "entity_type" : "TIME" }, "6":{ "entity_type" : "DATE" }, "7":{ "entity_type" : "ORG" } } # 实体标签 在以下表格中，我们列出了本数据集包含的所有实体标签，包括每个标签的说明与示例。 | 实体类别标签 | 说明 | 示例 | | --------------------- | ----------- | -------- | | PERSON | 人物姓名，含虚构人物 | 海格（Hagrid）、杰森·奇科拉（Jason Chicola）、W·E·B·杜波依斯（W. E. B. Du Bois） | | NORP | 国籍、宗教或政治团体 | 美国人（American）、中国人（Chinese）、共和党（Republican）、美国大老党（Grand Old Party）、罗马天主教徒（Roman Catholic） | | FAC | 建筑物、机场、高速公路、桥梁等设施 | 金门大桥（Golden Gate Bridge）、帝国大厦（the Empire State Building） | | ORG | 公司、机构、组织等 | Rev、通用汽车（General Motors）、美国证券交易委员会（SEC）、美国全国有色人种协进会（NAACP） | | GPE | 国家、城市、州等地缘政治实体 | 意大利（Italy）、美国（US）、波士顿（Boston）、新西兰（New Zealand） | | LOC | 非GPE的地理位置，如山脉、水域等 | 北部（the North）、落基山脉（the Rocky Mountains） | | PRODUCT | 物品、交通工具、食品等（不含服务） | 凯美瑞（Camry）、舒芬太尼（Sufentanil）、ARX-02 | | EVENT | 命名的飓风、战役、战争、体育赛事等 | 新冠疫情（COVID-19）、西班牙大流感（the Spanish Flu）、卡特里娜飓风（Hurricane Katrina）、第二次世界大战（World War II） | | WORK_OF_ART | 书籍、歌曲等作品名称 | 《弗兰肯斯坦》（Frankenstein）、《蒙娜丽莎》（The Mona Lisa） | | LAW | 正式颁布的法律文件等 | 1792年《福克斯法案》（Fox’s Act 1792）、美国宪法第五条（Article 5 of the Constitution） | | LANGUAGE | 命名语言 | 世界语（Esperanto）、西班牙语（Spanish）、斯瓦希里语（Swahili） | | DATE | 绝对/相对日期或时间段 | 第一季度（Q1）、去年年底（the end of last year） | | TIME | 单日以内的时间单位 | 早晨（Morning）、30分钟（30 minutes） | | PERCENT | 百分比，含`%`符号 | 约10%（Approximately 10%）、5% | | MONEY | 货币价值，含货币单位 | 400亿美元（$40 billion）、2万英镑（20 thousand pounds） | | QUANTITY | 度量单位，如重量、距离等 | 10公里（10 kilometers）、约80吨（approximately 80 tons） | | ORDINAL | 序数词，如第一、第二等 | 第三（Third）、第八（eighth）、第4（4th） | | CARDINAL | 不属于其他类别的数字 | 20、50、1420 | | ABBREVIATION | 全大写的缩写词，每个字母代表一个单词，即首字母缩略词 | AFK（away from keyboard）、限制性股票单位（RSU, restricted stock unit）、联邦紧急事务管理局（FEMA） | | WEBSITE | 完整网址 | www.rev.com、indeed.com | | YEAR | 代表年份的4位数字 | 2020、2021 | | CONTRACTION | 缩略词，由原词缩短而来或可转换为通用形式 | I’ll（I will）、going to（gonna） | | ALPHANUMERIC | 包含字母和数字的Token | 8th、Q4 | | FALLBACK | 不符合常规词汇的词语，通常包含未知符号（如`&`）或未完整说出的词语（如`th-`） | Q&M、lo- | # 实验结果 论文中收录的实验结果及所有实体类别的词错误率（Word Error Rate, WER）均可在`transcripts`目录下查看。 ## Eval-10：代表性Earnings21子集 除论文中收录的实验结果外，我们还提供了名为Eval-10的子集，该子集为完整Earnings21语料库中抽取的10小时代表性样本。该子集并非用于替代完整数据集，而是为研究人员提供快速验证其系统性能的途径，以便在完整数据集上开展正式实验。该子集上所有系统的WER计算结果均可在`transcripts`目录下的同一表格中查看。 # 词错误率计算 本数据集的所有分析均通过我们最新发布的[fstalign](https://github.com/revdotcom/fstalign/tree/master)工具完成。我们强烈推荐使用该工具以快速上手Earnings21数据集的使用。 # 数据集引用 本数据集相关研究已提交至Interspeech 2021会议。描述本研究方法与实验结果的论文可在arXiv平台获取：https://arxiv.org/pdf/2104.11348.pdf @misc{delrio2021earnings21, title={Earnings-21: A Practical Benchmark for ASR in the Wild}, author={Miguel Del Rio and Natalie Delworth and Ryan Westerman and Michelle Huang and Nishchal Bhandari and Joseph Palakapilly and Quinten McNamara and Joshua Dong and Piotr Zelasko and Miguel Jette}, year={2021}, eprint={2104.11348}, archivePrefix={arXiv}, primaryClass={cs.CL} }