TeraflopAI/SEC-EDGAR

--- license: apache-2.0 task_categories: - text-generation - text-classification language: - en tags: - finance - edgar - sec size_categories: - 1M<n<10M --- [Datamule](https://datamule.xyz/), [Teraflop AI](https://www.teraflopai.com/), and [Eventual](https://www.eventual.ai/) collaborated to release the SEC-EDGAR dataset.  The dataset contains 590 gbs of data, spanning 8 million samples and 43 billion tokens from all major filings in the SEC EDGAR database. The bulk data was collected using [datamule-python](https://github.com/john-friedman/datamule-python) library and the official [datamule api](https://datamule.xyz/) created by [John Friedman](https://john-friedman.github.io/). The datamule Python library is a package for collecting, manipulating, and processing the SEC Edgar data at scale. Datamule provides a simple open-source api interface to easily download each of a company's filings by ticker and submission type. SEC EDGAR rate limits at 10 requests per second. Constantly crawling 8 million major filings without network overhead takes over 10 days alone, following the official EDGAR guidance. The documentation for datamule can be found [here](https://john-friedman.github.io/datamule-python/). The dataset contains the raw contents of each major filing, the extracted and parsed HTML/XML plaintext, and relevant metadata such as the filing’s accession number, filing date, period, documents, and filer. The raw document contents are provided so that you may use your own custom parser to extract the HTML/XML to plaintext. The text was parsed and extracted from the HTML/XML contents using the [selectolax](https://selectolax.readthedocs.io/en/latest/index.html) HTML parser and a modified version of [doc2dict](https://github.com/john-friedman/doc2dict/tree/main) and [secsgml](https://github.com/john-friedman/secsgml) libraries. The SEC SGML library is used to parse through the [Standard Generalized Markup Language](https://en.wikipedia.org/wiki/Standard_Generalized_Markup_Language) document format used by the Securities and Exchange Commission and to handle [daily archive](http://sec.gov/Archives/edgar/Feed/) and [submission file types](https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/0000950170-22-000796.txt). The doc2dict library provides multiple parsers for extracting HTML, XML, and PDF content, and was used to convert to plaintext and explicitly handle table mappings. The documentation for [doc2dict](https://john-friedman.github.io/doc2dict/whitepaper/) can be found here. A total of 8 million individual filings were extracted with metadata. The document metadata contains the file type, sequence, filename, description, and number of SEC SGML bytes. The filer metadata contains the company name, Central Index Key, assigned Standard Industrial Classification Codes, IRS number, state of incorporation, fiscal year, act, file number, business address, and other relevant information. Samples per document type: | Filing | Total number of samples | | :---- | :---- | | Form 5 | 114,724 | | Form 4 | 4,474,981 | | Form 3 | 387,465 | | S-1 | 24,866 | | S-8 | 95,543 | | 10-K | 223,275 | | 8-K | 1,952,207 | | 20-F | 19,428 | | 10-Q | 674,240 | | 144 | 88,726 | | Total | 8,055,455 | To collect the total token counts of each filing, we used the [Comma v0.1 tokenizer](https://huggingface.co/common-pile/comma-v0.1-1t), a BPE-based tokenizer with a vocabulary size of 64,000. The dataset encompasses a total of 43 billion clean tokens for training LLMs and building retrieval pipelines.  Total token counts for each filing: | Filing | Total token count | | :---- | :---- | | 10-K | 14,518,876,137 | | 20-F | 2,917,164,397 | | Form 5 | 66,330,315 | | Form 4 | 1,676,565,503 | | Form 3 | 110,098,014 | | 10-Q | 17,509,723,617 | | S-1 | 2,914,107,827 | | S-8 | 472,867,864 | | 8-K | 3,466,866,649 | | 144 | 73,218,304 | | Total | 43,725,818,627 | We are building open-source state-of-the-art search across numerous domains. If you would like to help support or contribute to future open-source projects and dataset releases, you can join our [Discord](https://discord.gg/bWW8Wbhxhx) or contact us directly [here](https://x.com/EnricoShippole). You can use Teraflop AI segmentation, embedding, and search APIs today for free. Sign up for the Teraflop AI API platform [here](https://platform.teraflopai.com/signup).许可证：Apache-2.0 任务类别： - 文本生成 - 文本分类 语言： - 英语 标签： - 金融 - EDGAR - SEC 样本规模： - 100万 < 样本数 < 1000万 [Datamule](https://datamule.xyz/)、[Teraflop AI](https://www.teraflopai.com/)与[Eventual](https://www.eventual.ai/)合作发布了SEC EDGAR数据集。  该数据集包含590GB数据，涵盖美国证券交易委员会（SEC）电子数据收集、分析与检索系统（EDGAR）数据库中所有主要申报文件的800万份样本与430亿个Token。 批量数据通过[datamule-python](https://github.com/john-friedman/datamule-python)库与由[John Friedman](https://john-friedman.github.io/)开发的官方[datamule API](https://datamule.xyz/)收集。datamule Python库是一款用于大规模收集、处理与操作SEC EDGAR数据的开源包，提供简洁的API接口，可按股票代码与申报类型轻松下载每家公司的各类申报文件。根据SEC EDGAR的官方规则，其请求速率限制为每秒10次，按照官方指南，仅无网络开销地爬取800万份主要申报文件就需要超过10天。datamule的文档可参见[此处](https://john-friedman.github.io/datamule-python/)。 该数据集包含每份主要申报文件的原始内容、提取并解析后的HTML/XML纯文本，以及相关元数据，例如申报文件的接入号、申报日期、报告期、文档列表与申报方信息。提供原始文档内容以便用户使用自定义解析器将HTML/XML转换为纯文本。文本通过[selectolax](https://selectolax.readthedocs.io/en/latest/index.html) HTML解析器以及修改版的[doc2dict](https://github.com/john-friedman/doc2dict/tree/main)与[secsgml](https://github.com/john-friedman/secsgml)库从HTML/XML内容中解析提取。 SEC SGML库用于解析美国证券交易委员会（SEC）采用的标准通用标记语言（Standard Generalized Markup Language, SGML）文档格式，并处理[每日归档文件](http://sec.gov/Archives/edgar/Feed/)与[提交文件类型](https://www.sec.gov/Archives/edgar/data/1318605/000095017022000796/0000950170-22-000796.txt)。doc2dict库提供多种解析器用于提取HTML、XML与PDF内容，用于转换为纯文本并显式处理表格映射，其文档可参见[此处](https://john-friedman.github.io/doc2dict/whitepaper/)。 本次共提取了800万份带元数据的单独申报文件。文档元数据包含文件类型、序号、文件名、描述与SEC SGML字节数。申报方元数据包含公司名称、中央索引键（Central Index Key, CIK）、标准行业分类代码、国税局编号（Internal Revenue Service, IRS）、注册州、会计年度、适用法案、文件编号、办公地址及其他相关信息。 各申报类型样本分布： | 申报类型 | 样本总数 | | :---- | :---- | | 5号表格 | 114,724 | | 4号表格 | 4,474,981 | | 3号表格 | 387,465 | | S-1表格 | 24,866 | | S-8表格 | 95,543 | | 10-K表格 | 223,275 | | 8-K表格 | 1,952,207 | | 20-F表格 | 19,428 | | 10-Q表格 | 674,240 | | 144表格 | 88,726 | | 总计 | 8,055,455 | 为统计每份申报文件的总Token数，我们使用了[Comma v0.1分词器](https://huggingface.co/common-pile/comma-v0.1-1t)，这是一款基于字节对编码（Byte Pair Encoding, BPE）的分词器，词汇表大小为64,000。该数据集总计包含430亿个干净Token，可用于训练大语言模型（Large Language Model, LLM）与构建检索管线。  各申报类型总Token数统计： | 申报类型 | 总Token数 | | :---- | :---- | | 10-K表格 | 14,518,876,137 | | 20-F表格 | 2,917,164,397 | | 5号表格 | 66,330,315 | | 4号表格 | 1,676,565,503 | | 3号表格 | 110,098,014 | | 10-Q表格 | 17,509,723,617 | | S-1表格 | 2,914,107,827 | | S-8表格 | 472,867,864 | | 8-K表格 | 3,466,866,649 | | 144表格 | 73,218,304 | | 总计 | 43,725,818,627 | 我们正在多个领域构建开源的前沿搜索系统。如果您希望支持或为未来的开源项目和数据集发布做出贡献，可以加入我们的[Discord社区](https://discord.gg/bWW8Wbhxhx)或通过[此处](https://x.com/EnricoShippole)直接联系我们。 您现在可以免费使用Teraflop AI的文本分割、嵌入与搜索API，前往[Teraflop AI API平台](https://platform.teraflopai.com/signup)注册即可。