ArtificialAnalysis/Earnings22-Cleaned-AA

--- license: apache-2.0 task_categories: - automatic-speech-recognition language: - en tags: - speech-to-text - word-error-rate - benchmark - cleaned-transcripts - earnings22 pretty_name: Earnings22-Cleaned-AA size_categories: - n<1K dataset_info: features: - name: id dtype: string - name: duration dtype: float64 - name: transcript dtype: string - name: language dtype: string - name: url dtype: string - name: dataset dtype: string - name: file_name dtype: string splits: - name: test num_examples: 6 configs: - config_name: default data_files: - split: test path: earnings22_cleaned_aa_v1.jsonl source_datasets: - esb/datasets --- # Earnings22-Cleaned-AA **Quick links:** [AA Speech-to-Text Leaderboard](https://artificialanalysis.ai/speech-to-text) | [AA-WER v2.0 article](https://artificialanalysis.ai/articles/aa-wer-v2) Earnings22-Cleaned-AA is a cleaned subset of the English Earnings-22 test data from [esb/datasets](https://huggingface.co/datasets/esb/datasets), a corpus of corporate earnings calls from global companies with speakers of many different nationalities and accents. This cleaned subset is the Earnings-22 portion included in AA-WER v2. We manually reviewed and corrected errors in the original ground-truth transcriptions to ensure fairer evaluation of Speech to Text (STT) models. This dataset is part of [AA-WER v2.0](https://artificialanalysis.ai/articles/aa-wer-v2), the Speech to Text accuracy benchmark by [Artificial Analysis](https://artificialanalysis.ai/speech-to-text), where it carries a 25% weighting alongside AA-AgentTalk (50%) and VoxPopuli-Cleaned-AA (25%). ## Dataset Summary | Property | Value | | --- | --- | | Source | Subset of [Earnings-22](https://huggingface.co/datasets/esb/datasets) (ESB) English test split | | Domain | Corporate earnings calls | | Number of samples | 6 | | Sample duration range | ~14–22 minutes | | Total duration | ~115 minutes | | Language | English | ## Motivation for Correction Reference transcripts in the original Earnings22 test set contained inaccuracies — instances where the ground truth didn't match what was actually spoken. Inaccurate ground truth penalizes models that correctly transcribe the audio, inflating WER scores unfairly. On average, model WER on Earnings22 went down 5.6 percentage points (p.p.) after cleaning, and no models had higher WER after cleaning ([article](https://artificialanalysis.ai/articles/aa-wer-v2)).  ## Dataset Correction We corrected transcripts to reflect verbatim what speakers said. Key corrections included: - **Incorrect words:** Misspellings, misheard words, incorrect contractions in the original references - **Missed words:** Retained or added repetitions for verbatim accuracy (e.g., "the the" where the speaker genuinely repeated a word) - **Partial stuttering:** Removed incomplete word fragments (e.g., "evac-" in "evac- evacuate") as these are inherently ambiguous in transcription - **Grammar and tense:** When speakers used incorrect grammar (particularly speakers with accents) but the word choice was clear, we kept verbatim words as spoken rather than correcting them Elements already normalized by the [Whisper normalizer](https://github.com/openai/whisper/tree/main/whisper/normalizers) package (e.g., capitalization, punctuation, and filler words) were not modified, since these differences are already handled during WER calculation. ### Sample <audio controls><source src="https://huggingface.co/datasets/ArtificialAnalysis/Earnings22-Cleaned-AA/resolve/main/audio/03c13d67-c016-4a2b-91d8-e013d5d6bd2b.mp3" type="audio/mp3" /></audio> > Thank you, Darcy, and welcome everyone to our December quarterly analyst call. December quarterly production showed a considerable improvement on the September quarter with record production throughput and improving grades, improving recoveries and improving cash flow. Unfortunately, delays accessing higher grade parts of the open pit resulted in lower grades than projected in our guidance. On the exploration front, today we announced a 70% increase in our 100% owned Yamarna resources. So they now sit at 0.5 million ounces... ## Usage ```python from datasets import load_dataset dataset = load_dataset("ArtificialAnalysis/Earnings22-Cleaned-AA", split="test") ``` `url` fields in the dataset point to repo-local audio files under `audio/`. ## WER Evaluation For WER evaluation, we use the [jiwer](https://github.com/jitsi/jiwer) library with a custom text normalizer building on [OpenAI's Whisper normalizer](https://github.com/openai/whisper/tree/main/whisper/normalizers). Our normalizer adds: - Digit splitting to prevent number grouping mismatches (e.g., "1405 553 272" vs. "1405553272") - Preservation of leading zeros in codes and identifiers - Normalization of spoken symbols (e.g., "+", "_") - Stripping redundant ":00" in times (e.g., "7:00pm" vs. "7pm") - Additional US/UK English spelling equivalences (e.g., "totalled" vs. "totaled") - Accepted equivalent spellings for ambiguous proper nouns (e.g., "Mateo" vs. "Matteo") Results within the dataset are aggregated as an audio-duration-weighted average WER so that numerous short clips do not bias results compared to longer files. ## Citation If you use this dataset, please cite: ```bibtex @misc{artificialanalysis2026earnings22cleaned, title={Earnings22-Cleaned-AA: Cleaned Ground Truth Transcripts for Earnings22 English Test Set}, author={Artificial Analysis}, year={2026}, url={https://artificialanalysis.ai/articles/aa-wer-v2} } ``` ## Resources - [Full results and leaderboard](https://artificialanalysis.ai/speech-to-text) - [Benchmarking methodology](https://artificialanalysis.ai/speech-to-text/methodology) - [AA-WER v2.0 article](https://artificialanalysis.ai/articles/aa-wer-v2) - [VoxPopuli-Cleaned-AA on Hugging Face](https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA) ## Versioning Current version: 1.0 Used in: AA-WER v2.0 benchmark release Specific dataset versions used for each AA-WER release are documented in the [Artificial Analysis methodology](https://artificialanalysis.ai/speech-to-text/methodology). ## License This dataset is released under Apache-2.0. For upstream terms, see [esb/datasets](https://huggingface.co/datasets/esb/datasets). ## Feedback These cleaned transcripts reflect our best effort at verbatim ground truth, informed by manual review and cross-validation. Future refinements will be released as subsequent versions (v2+). If you spot issues, we welcome feedback via our [contact page](https://artificialanalysis.ai/contact) or [Discord](https://discord.com/channels/1459307953394810952/1460789635239121162).许可证：Apache-2.0 任务类别： - 自动语音识别（automatic-speech-recognition） 语言： - 英语 标签： - 语音转文字（speech-to-text） - 词错误率（word-error-rate） - 基准测试（benchmark） - 清理后转录本（cleaned-transcripts） - earnings22 展示名称：Earnings22-Cleaned-AA 规模类别： - 样本数少于1000（n<1K） 数据集信息： 特征： - 名称：id，数据类型：字符串（string） - 名称：duration，数据类型：双精度浮点数（float64） - 名称：transcript，数据类型：字符串（string） - 名称：language，数据类型：字符串（string） - 名称：url，数据类型：字符串（string） - 名称：dataset，数据类型：字符串（string） - 名称：file_name，数据类型：字符串（string） 数据划分： - 名称：测试集（test），样本数量：6 配置项： - 配置名称：默认配置（default） 数据文件： - 数据划分：测试集（test） 路径：earnings22_cleaned_aa_v1.jsonl 来源数据集： - esb/datasets # Earnings22-Cleaned-AA **快速链接：** [AA 语音转文字排行榜](https://artificialanalysis.ai/speech-to-text) | [AA-WER v2.0 文章](https://artificialanalysis.ai/articles/aa-wer-v2) Earnings22-Cleaned-AA 是来自 [esb/datasets](https://huggingface.co/datasets/esb/datasets) 的英语版 Earnings-22 测试数据的清理子集，该数据集是包含全球多家企业的公司财报电话会议语料库，与会发言者涵盖不同国籍与口音。此清理子集为 AA-WER v2 中收录的 Earnings-22 部分。我们对原始标准转录文本进行了人工审核与错误修正，以确保对语音转文字（Speech to Text, STT）模型的评估更为公平。 该数据集是由 [Artificial Analysis](https://artificialanalysis.ai/speech-to-text) 推出的语音转文字准确率基准测试 [AA-WER v2.0](https://artificialanalysis.ai/articles/aa-wer-v2) 的组成部分，在该基准中其权重为25%，其余权重分别由 AA-AgentTalk（50%）与 VoxPopuli-Cleaned-AA（25%）承担。 ## 数据集概述 | 属性 | 取值 | | --- | --- | | 来源 | [Earnings-22](https://huggingface.co/datasets/esb/datasets)（ESB）英语测试划分的子集 | | 领域 | 企业财报电话会议 | | 样本数量 | 6 | | 单样本时长范围 | 约14–22分钟 | | 总时长 | 约115分钟 | | 语言 | 英语 | ## 修正动机 原始Earnings22测试集的参考转录本存在不准确之处：即标准文本与实际发言内容不匹配的情况。不准确的标准文本会惩罚那些正确转录音频的模型，导致词错误率（Word Error Rate, WER）得分被不合理地高估。经清理后，Earnings22数据集上的模型WER平均下降了5.6个百分点（p.p.），且所有模型的WER均未出现上升（[相关文章](https://artificialanalysis.ai/articles/aa-wer-v2)）。  ## 数据集修正 我们对转录本进行修正以完全还原发言者的原话，主要修正内容包括： - **错误词汇**：原始参考文本中的拼写错误、误听词汇以及错误的缩略形式 - **漏录词汇**：为保证逐字还原的准确性，保留或补充了重复的发言（例如发言者确实重复了词汇“the the”的场景） - **半口吃片段**：移除了不完整的词汇片段（例如“evac- evacuate”中的“evac-”），因为此类片段在转录中本质上存在歧义 - **语法与时态**：当发言者出现语法错误（尤其是带口音的发言者）但词汇选择清晰时，我们保留原始发言的逐字内容，而非进行语法修正 已由 [Whisper 标准化工具](https://github.com/openai/whisper/tree/main/whisper/normalizers) 包完成标准化的内容（例如大小写、标点符号与填充词）未被修改，因为此类差异在WER计算过程中已被处理。 ### 样本示例 <audio controls><source src="https://huggingface.co/datasets/ArtificialAnalysis/Earnings22-Cleaned-AA/resolve/main/audio/03c13d67-c016-4a2b-91d8-e013d5d6bd2b.mp3" type="audio/mp3" /></audio> > 感谢达西，欢迎各位参加我们的12月季度分析师电话会议。12月季度的产量相较9月季度有显著提升，创下了产量吞吐量的新高，同时品位、回收率与现金流均有所改善。遗憾的是，由于进入露天矿高品位区域的进度延迟，最终品位未达我们的指导预期。在勘探方面，我们今日宣布对100%持股的Yamarna矿资源量提升70%，目前该矿资源量已达50万盎司…… ## 使用方法 python from datasets import load_dataset dataset = load_dataset("ArtificialAnalysis/Earnings22-Cleaned-AA", split="test") 数据集中的`url`字段指向仓库内`audio/`路径下的音频文件。 ## 词错误率（WER）评估 在WER评估中，我们使用 [jiwer](https://github.com/jitsi/jiwer) 库，并基于 [OpenAI Whisper 标准化工具](https://github.com/openai/whisper/tree/main/whisper/normalizers) 开发了自定义文本标准化器，该标准化器新增以下功能： - 数字拆分：防止数字分组不匹配（例如“1405 553 272”与“1405553272”的差异） - 保留代码与标识符中的前导零 - 口语符号标准化（例如“+”“_”） - 移除时间格式中冗余的“:00”（例如“7:00pm”与“7pm”） - 新增英美英语拼写等价转换（例如“totalled”与“totaled”） - 对有歧义的专有名词使用公认的等价拼写（例如“Mateo”与“Matteo”） 数据集内的评估结果以音频时长加权平均WER进行汇总，以避免大量短音频片段相较于长音频片段对结果产生偏差。 ## 引用方式 若您使用该数据集，请引用以下文献： bibtex @misc{artificialanalysis2026earnings22cleaned, title={Earnings22-Cleaned-AA: Cleaned Ground Truth Transcripts for Earnings22 English Test Set}, author={Artificial Analysis}, year={2026}, url={"https://artificialanalysis.ai/articles/aa-wer-v2"} } ## 相关资源 - [完整结果与排行榜](https://artificialanalysis.ai/speech-to-text) - [基准测试方法论](https://artificialanalysis.ai/speech-to-text/methodology) - [AA-WER v2.0 文章](https://artificialanalysis.ai/articles/aa-wer-v2) - [Hugging Face 上的 VoxPopuli-Cleaned-AA 数据集](https://huggingface.co/datasets/ArtificialAnalysis/VoxPopuli-Cleaned-AA) ## 版本说明 当前版本：1.0 应用场景：AA-WER v2.0 基准测试发布版 各AA-WER版本所使用的具体数据集版本已在 [Artificial Analysis 方法论文档](https://artificialanalysis.ai/speech-to-text/methodology) 中说明。 ## 许可证 本数据集采用 Apache-2.0 许可证发布。上游数据集的相关条款请参阅 [esb/datasets](https://huggingface.co/datasets/esb/datasets)。 ## 反馈 本清理后的转录本是我们通过人工审核与交叉验证所得到的逐字标准文本的最优版本。后续优化版本将以v2及以上版本形式发布。若您发现任何问题，欢迎通过 [联系页面](https://artificialanalysis.ai/contact) 或 [Discord社区](https://discord.com/channels/1459307953394810952/1460789635239121162) 向我们反馈。