aikathata/indonesian-dfk-cpt-dataset

该数据集是一个印尼语文本语料库，专为继续预训练（CPT）而设计，特别关注DFK域，即常涉及虚假信息、诽谤和仇恨言论的主题领域，如政治/政府、法律/犯罪、宗教、敏感社会问题、公共卫生、冲突/安全等。数据集不包含实际的有害内容，而是针对这些主题的文本。它包括正式数据（来自新闻、维基百科和期刊）和非正式合成数据（通过重写正式文本以模拟印尼社交媒体风格生成），旨在帮助模型理解印尼语的语言变体，包括正式和非正式表达。数据集以JSON Lines格式存储，分为DFK域和通用域，并提供了详细的来源和组成统计，适用于印尼语NLP模型开发和研究。

This dataset is an Indonesian text corpus designed for Continued Pre-Training (CPT), focusing on the DFK domain, which refers to topics frequently targeted by disinformation, defamation, and hate speech in Indonesia, such as politics/government, law/crime, religion, sensitive social issues, public health, conflict/security, etc. The corpus does not contain actual harmful content but includes texts related to these themes. It comprises formal data (from news, Wikipedia, and journals) and non-formal synthetic data (generated by rewriting formal texts to mimic Indonesian social media style), aiming to help models understand linguistic variations in Indonesian, including both formal and informal expressions. The dataset is stored in JSON Lines format, divided into DFK and general domains, with detailed source and composition statistics, suitable for Indonesian NLP model development and research.