kwassl-ai/noah-text-10BT

--- license: odc-by task_categories: - text-generation language: - en pretty_name: NoAH Text 10BT --- # Dataset Card for NoAH Text First 10B Tokens of [NoAH Text](https://huggingface.co/datasets/kwassl-ai/noah-text) ## Dataset Details ### Dataset Description <!-- Provide a longer summary of what this dataset is. --> NoAH Text is a curated dataset of English spoken and written text, designed for training language models without attribution requirements. It consists of public domain works and content released under permissive licenses: **CC0**, **MIT-0**, and **The Unlicense**. By aggregating and filtering the Common Corpus, NoAH provides a clean, legally unambiguous resource for research and development in natural language processing. This dataset is a sample of 10BT tokens from the full NoAH Text dataset. - **Language(s) (NLP):** English - **License:** [Open Data Commons Attribution License](https://opendatacommons.org/licenses/by/) ## Uses ## Dataset Structure Dataset Fields: - `identifier`: unique text identifier. In many cases, this is also the link to the original resources. - `collection`: name of one of the XX sub-collections curated for Common corpus. - `open type`: one of the six leading collection groupings: - `license`: sharing rights for the content either uncopyrighted (public domain, US federal public domain, CC0 on Wikidata) or various free licenses (Creative Commons, MIT, French Licence ouverte, etc.) - `date`: date of creation of the resource where known. Due to the significance of public domain and other cultural heritage content, more than half of Common Corpus predates the 21st century. - `title`: title of the resource when known or alternatively the filename. - `creator`: institution publishing/collecting/curating the resource. - `language`: automatically identified language. - `word_count`: number of space delimited words. - `token_count`: number of tokens as calculated by Pleias official tokenizer. - `text`: full text, without formatting. ## Dataset Creation ### Curation Rationale With NoAH, we try to start addressing the growing need for high-quality, legally unambiguous text corpora in the NLP community. Many existing datasets are encumbered by complex licensing terms, creating barriers for researchers and developers who require clear, attribution-free resources. By focusing exclusively on public domain works and permissive licenses (CC0, MIT-0, and The Unlicense), NoAH eliminates legal uncertainty and simplifies compliance, enabling unrestricted use in training language models. ### Source Data Contains information from [Common Corpus](https://huggingface.co/datasets/PleIAs/common_corpus) by PleIAs. #### Personal and Sensitive Information Beyond filtering by license and language type, we have not processed the data any further. For this reason, any personally identifiable information (PII) present in the original dataset may still be present in this dataset. ## Bias, Risks, and Limitations Beyond filtering by license and language type, we have not processed the data any further. For this reason, any bias present in the original dataset may still be present in this dataset.license: 开放数据通用公共署名牌证（Open Data Commons Attribution License，缩写ODC-BY） task_categories: - 文本生成（Text Generation） language: - 英语（en） pretty_name: NoAH Text 10BT --- # NoAH Text 数据集卡片 本数据集为[NoAH Text](https://huggingface.co/datasets/kwassl-ai/noah-text)完整数据集的前100亿Token（Token）。 ## 数据集详情 ### 数据集描述 <!-- 提供该数据集的详细概述。 --> NoAH Text是一套经过精心遴选的英语口语与书面文本数据集，专为无需署名要求的大语言模型（Large Language Model，LLM）训练打造。该数据集包含公有领域作品，以及采用宽松许可证发布的内容：**CC0**、**MIT-0**与**The Unlicense**。通过对通用语料库（Common Corpus）进行聚合与筛选，NoAH可为自然语言处理领域的研发工作提供一套清晰合规、无法律歧义的优质资源。 本数据集为完整NoAH Text数据集的100亿Token采样子集。 - **自然语言处理所用语言：** 英语 - **许可证：** [开放数据通用公共署名牌证（Open Data Commons Attribution License）](https://opendatacommons.org/licenses/by/) ## 用途 ## 数据集结构 ### 数据集字段 - `identifier`（标识符）：文本唯一标识符。多数情况下，该字段同时指向原始资源的链接地址。 - `collection`（语料子集合名称）：为通用语料库精选的XX个子集合之一的名称。 - `open type`（开放类型）：六大主流语料集合分组之一： - `license`（许可证）：内容的共享权限，分为无版权类型（公有领域、美国联邦政府公有领域、维基数据上的CC0）与各类免费许可证（知识共享、MIT、法国开放许可证等）。 - `date`（创建日期）：已知的资源创建时间。鉴于公有领域与其他文化遗产内容占比颇高，通用语料库中超过一半的资源早于21世纪。 - `title`（标题）：已知的资源标题，若无则使用文件名替代。 - `creator`（发布/收集/精选机构）：发布、收集或精选该资源的机构。 - `language`（语言）：自动识别出的文本语言。 - `word_count`（单词数）：以空格分隔的单词总数。 - `token_count`（Token数）：通过Pleias官方分词器（Tokenizer）计算得到的Token总数。 - `text`（文本内容）：未经格式处理的完整文本。 ## 数据集构建 ### 遴选依据 针对自然语言处理社区对高质量、无法律歧义文本语料库日益增长的需求，NoAH旨在填补这一空白。当前多数现有数据集受限于复杂的许可条款，为需要清晰无署名要求资源的研究人员与开发者设置了壁垒。通过仅采用公有领域作品与宽松许可证（CC0、MIT-0及The Unlicense），NoAH消除了法律不确定性，简化了合规流程，可无限制用于大语言模型训练。 ### 源数据 本数据集包含PleIAs发布的[通用语料库（Common Corpus）](https://huggingface.co/datasets/PleIAs/common_corpus)中的相关信息。 #### 个人与敏感信息 除按许可证与语言类型进行筛选外，我们未对数据进行进一步处理。因此，原始数据集中存在的任何个人可识别信息（Personally Identifiable Information，PII）仍可能保留在本数据集中。 ## 偏差、风险与局限性 除按许可证与语言类型进行筛选外，我们未对数据进行进一步处理。因此，原始数据集中存在的任何偏差仍可能保留在本数据集中。