Minuri/nsina_cleaned_version

--- language: - si license: cc-by-sa-4.0 task_categories: - text-generation pretty_name: Sinhala Cleaned Sentences - NSINA size_categories: - 1M<n<10M tags: - sinhala - low-resource - pretraining - news - cleaned - deduplicated --- # Sinhala Cleaned Sentences - NSINA Cleaned and deduplicated Sinhala sentences derived from `Minuri/nsina-sentences-raw`, produced through a multi-stage cleaning pipeline. This repo was used as pipeline storage across cleaning stages, with the final output being `stage10_final_corpus_deduped.csv`. ## Final Output | File | Rows | Description | |---|---|---| | `stage10_final_corpus_deduped.csv` | 3,546,626 | Final cleaned and deduplicated sentences | ## Dataset Structure (final output) | Column | Description | |---|---| | `sentence` | Cleaned Sinhala sentence | ## Pipeline Stages This repo stores intermediate CSV files across the 12-stage cleaning pipeline (stage2 through stage10), including cleaning logs and filter reports. ## Pipeline Position `Minuri/nsina-sentences-raw` → **this repo** → `Minuri/diverse_sinhala_dataset` ## Sources & Licenses | Source | License | |---|---| | [sinhala-nlp/NSINA](https://huggingface.co/datasets/sinhala-nlp/NSINA) | CC BY-SA 4.0 - ShareAlike applies | ## Related Repositories | Repo | Description | |---|---| | `Minuri/nsina-sentences-raw` | Raw sentences (5,415,583) | | `Minuri/diverse_sinhala_dataset` | Final parent corpus |language: - 僧伽罗语（Sinhala） license: CC-BY-SA-4.0 task_categories: - 文本生成 pretty_name: 僧伽罗语清洗语句数据集 - NSINA size_categories: - 100万<样本量<1000万 tags: - 僧伽罗语 - 低资源语言 - 预训练 - 新闻 - 清洗后数据 - 去重数据 --- # 僧伽罗语清洗语句数据集 - NSINA 本数据集源自`Minuri/nsina-sentences-raw`，经多阶段清洗流程处理后生成清洗且去重的僧伽罗语语句集合。本仓库用于存储各清洗阶段的中间数据，最终输出文件为`stage10_final_corpus_deduped.csv`。 ## 最终输出文件 | 文件 | 行数 | 描述 | |---|---|---| | `stage10_final_corpus_deduped.csv` | 3,546,626 | 最终清洗并去重的语句集 | ## 数据集结构（最终输出） | 列名 | 描述 | |---|---| | `sentence` | 清洗后的僧伽罗语语句 | ## 清洗流程阶段 本仓库存储了12阶段清洗流程（阶段2至阶段10）的中间CSV文件，包含清洗日志与过滤报告。 ## 数据集流转路径 `Minuri/nsina-sentences-raw` → **本仓库** → `Minuri/diverse_sinhala_dataset` ## 数据源与许可证 | 数据源 | 许可证 | |---|---| | [sinhala-nlp/NSINA](https://huggingface.co/datasets/sinhala-nlp/NSINA) | CC BY-SA 4.0 - 适用知识共享署名-相同方式共享协议 | ## 相关仓库 | 仓库 | 描述 | |---|---| | `Minuri/nsina-sentences-raw` | 原始语句集（共5,415,583条） | | `Minuri/diverse_sinhala_dataset` | 最终父级语料库 |