NoeFlandre/herero-dataset

--- license: other license_name: multiple-licenses pretty_name: Herero Language Dataset language: - hz task_categories: - text-classification --- # Herero Language Dataset [](https://github.com/NoeFlandre/herero-dataset) [](https://huggingface.co/datasets/NoeFlandre/herero-dataset) A **multi-license collection** of [Herero (Otjiherero)](https://en.wikipedia.org/wiki/Herero_language) text for NLP research. This is **not** a derivative work—sources are aggregated with original licenses preserved. > **📦 Data:** This page — download Parquet files here > **💻 Code:** [GitHub Repository](https://github.com/NoeFlandre/herero-dataset) — scrapers, scripts, and documentation **⚠️ WARNING: This repository is a redistribution of materials from multiple upstream sources under different licenses. Each example remains subject to its original source license. Users are responsible for checking per-row licensing before reuse.** ## Dataset Summary | Metric | Value | |--------|-------| | Documents | 1,951 | | Words | 870,941 | | Sources | 7 | | Language | Herero (hz) | ## Sources | Source | Docs | Words | License | |--------|-----:|------:|---------| | FineWeb-2 | 1,272 | 672,378 | ODC-By 1.0 | | Omnilingual ASR | 437 | 60,485 | CC BY 4.0 | | GlotCC-V1 | 20 | 75,982 | CC BY 4.0 | | Herero Bible (1849) | 81 | 26,807 | Public Domain | | Storybooks Namibia | 46 | 13,246 | CC BY 4.0 | | FinePDFs | 25 | 17,800 | ODC-By 1.0 | | Wikipedia Incubator | 70 | 4,243 | CC BY-SA 3.0 | ## Data Splits | Split | Documents | % | |-------|-----------|---| | train | 1,761 | 90% | | validation | 95 | 5% | | test | 95 | 5% | ## Schema Each row includes provenance columns: | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique document ID | | `text` | string | Herero text (NFC normalized) | | `source` | string | Data source name | | `original_license` | string | Source license (check this!) | | `original_url` | string | Link to original source | | `changes_made` | string | Transformations applied | | `license_type` | string | License category | | `url` | string | Source URL | | `word_count` | int | Word count | | `char_count` | int | Character count | ## Usage ```python from datasets import load_dataset dataset = load_dataset("NoeFlandre/herero-dataset") # Check per-row license BEFORE use for example in dataset["train"]: print(f"License: {example['original_license']} - {example['source']}") ``` ## Collection Classification This dataset is a **collection**, not an adaptation: - Text is aggregated from sources without material modification - Original text is preserved (only formatting normalized) - No translation, rewriting, or merging across sources - Source boundaries maintained via `source` column ## Quality Assurance - **Deduplication**: Within-source SHA-256 content hashing - **Normalization**: Unicode NFC normalization (no text rewriting) - **Filtering**: Min-length 50 chars - **Reproducibility**: Fixed random seed (42) ## Known Limitations - ~871K words is moderate for low-resource language research - FineWeb-2 dominates (~77% of content) - Historical text uses 19th-century orthography - Speech transcripts may contain disfluencies ## Ethical Considerations - No personal data included - All content publicly available at time of scraping - Attribution preserved via `original_url` column - Users must respect individual source licenses ## Licensing Summary | License Type | Sources | Requirements | |-------------|---------|--------------| | Public Domain | 1 | None | | CC BY | 3 | Attribution | | CC BY-SA | 1 | Attribution + ShareAlike | | ODC-By | 2 | Attribution + Keep open | **You must check the `original_license` column for each example.** ## Files - `data/train.parquet` - Training split - `data/validation.parquet` - Validation split - `data/test.parquet` - Test split - `sources.csv` - Full source manifest ## Disclaimer This documentation is provided for informational purposes only and does not constitute legal advice. Uncertain licensing cases were reviewed conservatively. For legal questions, consult the original license texts or seek professional counsel.--- license: 其他 license_name: 多许可证 pretty_name: 赫雷罗语数据集 language: - hz task_categories: - 文本分类 --- # 赫雷罗语数据集 [](https://github.com/NoeFlandre/herero-dataset) [](https://huggingface.co/datasets/NoeFlandre/herero-dataset) 本数据集为面向自然语言处理（Natural Language Processing, NLP）研究的**多许可证合集**，收录[赫雷罗语（Otjiherero）](https://en.wikipedia.org/wiki/Herero_language)文本。本数据集并非衍生作品——所有来源均已聚合，且保留原始许可证。 > **📦 数据：** 本页面 — 可在此处下载Parquet文件 > **💻 代码：** [GitHub仓库](https://github.com/NoeFlandre/herero-dataset) — 包含爬取脚本、工具与文档 > **⚠️ 重要提示：** 本仓库为多个上游来源材料的再分发，各来源适用不同许可证。每条数据仍受其原始来源许可证约束。用户在复用前需自行检查每行数据的授权情况。 ## 数据集概览 | 指标 | 数值 | |--------|-------| | 文档数 | 1,951 | | 总词数 | 870,941 | | 来源数 | 7 | | 语言 | 赫雷罗语（hz） | ## 数据来源 | 来源 | 文档数 | 词数 | 许可证 | |--------|-----:|------:|---------| | FineWeb-2 | 1,272 | 672,378 | ODC-By 1.0 | | Omnilingual ASR | 437 | 60,485 | CC BY 4.0 | | GlotCC-V1 | 20 | 75,982 | CC BY 4.0 | | Herero Bible (1849) | 81 | 26,807 | 公有领域 | | Storybooks Namibia | 46 | 13,246 | CC BY 4.0 | | FinePDFs | 25 | 17,800 | ODC-By 1.0 | | Wikipedia Incubator | 70 | 4,243 | CC BY-SA 3.0 | ## 数据划分 | 划分集 | 文档数 | 占比 | |-------|-----------|---| | 训练集 | 1,761 | 90% | | 验证集 | 95 | 5% | | 测试集 | 95 | 5% | ## 数据字段结构 | 字段名 | 类型 | 描述 | |-------|------|-------------| | `id` | 字符串 | 唯一文档ID | | `text` | 字符串 | 赫雷罗语文本（已完成NFC标准化处理） | | `source` | 字符串 | 数据来源名称 | | `original_license` | 字符串 | 原始来源许可证（请务必核查此项！） | | `original_url` | 字符串 | 原始来源链接 | | `changes_made` | 字符串 | 所应用的转换操作 | | `license_type` | 字符串 | 许可证类别 | | `url` | 字符串 | 来源URL | | `word_count` | 整数 | 词数 | | `char_count` | 整数 | 字符数 | ## 使用示例 python from datasets import load_dataset dataset = load_dataset("NoeFlandre/herero-dataset") # 使用前请核查每行数据的许可证 for example in dataset["train"]: print(f"许可证：{example['original_license']} - {example['source']}") ## 数据集归类说明 本数据集为**合集**，而非改编作品： - 文本从各来源聚合，未进行实质性修改 - 保留原始文本（仅进行格式标准化） - 未进行跨来源的翻译、重写或合并 - 通过`source`字段保留数据来源边界 ## 质量保障措施 - 去重：基于来源内的SHA-256内容哈希 - 标准化：采用Unicode NFC格式标准化（未改写文本内容） - 过滤：仅保留字符数不少于50的文本 - 可复现性：使用固定随机种子（42） ## 已知局限性 - 约87.1万词的规模对于低资源语言研究而言属于中等体量 - FineWeb-2占比最高（约占总内容的77%） - 历史文本采用19世纪的正字法 - 语音转录文本可能包含语音停顿等不流畅内容 ## 伦理考量 - 未包含任何个人数据 - 所有内容在爬取时均为公开可获取的 - 通过`original_url`字段保留署名信息 - 用户需遵守各原始来源的许可证要求 ## 许可证汇总 | 许可证类型 | 来源数 | 合规要求 | |-------------|---------|--------------| | 公有领域 | 1 | 无 | | CC BY | 3 | 需注明原作者 | | CC BY-SA | 1 | 需注明原作者 + 相同方式共享 | | ODC-By | 2 | 需注明原作者 + 保持开源 | **请务必核查每条数据的`original_license`字段。** ## 文件清单 - `data/train.parquet` - 训练集划分文件 - `data/validation.parquet` - 验证集划分文件 - `data/test.parquet` - 测试集划分文件 - `sources.csv` - 完整来源清单 ## 免责声明 本文档仅用于提供信息参考，不构成法律建议。对于存在授权疑问的案例，已采用保守审查原则。如有法律疑问，请查阅原始许可证文本或咨询专业法律顾问。