neuralfoundry-coder/handiff-ko-corpus-v2

这个数据集是用于HanDiff韩语Diffusion LLM预训练的精选语料库。它包含多个来源的数据，如维基百科（kowiki）、学术论文、法律判例、法规修订文本、新闻正文、情感对话和领域对话（如餐厅/住宿/旅游等），总计约885,637个文档，总大小为3554.2 MB，格式为parquet（使用zstd-19压缩，约为原始JSONL的28%）。数据分片未直接包含，但会在训练时通过自动构建过程生成。

This dataset is a curated corpus for pre-training HanDiff Korean Diffusion LLM. It includes data from multiple sources such as Wikipedia (kowiki), academic papers, legal precedents, law amendments, news articles, emotional dialogues, and domain dialogues (e.g., restaurants/accommodations/tourism), totaling approximately 885,637 documents with a total size of 3554.2 MB, in parquet format (compressed with zstd-19, about 28% of raw JSONL). Data shards are not included directly but are generated automatically during training through a build process.