awidjaja/NLLB-id-neardup

该数据集是KoPI-NLLB的一个子集，专注于印度尼西亚各种语言的近重复条目。它以parquet格式提供，包含多种印度尼西亚的低资源语言。数据集包含文本数据及其元数据（url、score、source），适用于翻译、文本相似性和语言建模等自然语言处理任务。

This dataset is a subset of the KoPI-NLLB, focusing only on near-duplicate entries across various Indonesian languages. It is provided in parquet format and includes multiple low-resource languages from Indonesia. The dataset features textual data with accompanying metadata (url, score, source), making it useful for natural language processing tasks such as translation, text similarity, and language modeling.