Jeremydh911/SEC-EDGAR

SEC-EDGAR数据集是由Datamule、Teraflop AI和Eventual合作发布的金融领域数据集，包含来自美国证券交易委员会（SEC）EDGAR数据库中的所有主要文件。数据集总大小为590GB，涵盖800万样本和430亿个令牌。数据收集使用了datamule-python库和官方datamule api，处理过程包括HTML/XML解析和文本提取。数据集包含每个主要文件的原始内容、解析后的纯文本以及相关元数据。元数据包括文件类型、序列、文件名、描述等，以及公司名称、中央索引键、标准工业分类代码等。此外，数据集还提供了不同文件类型的样本数量和令牌数量统计，适用于训练大型语言模型和构建检索管道。

The SEC-EDGAR dataset is a financial domain dataset released in collaboration by Datamule, Teraflop AI, and Eventual, containing all major filings from the U.S. Securities and Exchange Commission (SEC) EDGAR database. The dataset totals 590GB, covering 8 million samples and 43 billion tokens. Data collection utilized the datamule-python library and the official datamule api, with processing including HTML/XML parsing and text extraction. The dataset contains the raw contents of each major filing, the parsed plaintext, and relevant metadata such as file type, sequence, filename, description, as well as company name, Central Index Key, Standard Industrial Classification Codes, etc. Additionally, the dataset provides statistics on the number of samples and tokens for different filing types, making it suitable for training large language models and building retrieval pipelines.