CUI03/german-commons

German Commons数据集是一个大规模、开放许可的德语文本数据集合，用于训练德语语言模型。该数据集汇集了来自41个不同来源的154.56亿个标记，涵盖35.78百万份文档，分布在7个主题领域：网络公共资源（Web Commons）、政治公共资源（Political Commons）、法律公共资源（Legal Commons）、新闻公共资源（News Commons）、经济公共资源（Economic Commons）、文化公共资源（Cultural Commons）和科学公共资源（Scientific Commons）。每个记录包含唯一标识符、来源数据集名称、主题子集、主文本内容、适用许可证列表、GPT-2标记计数、文本困惑度（使用KenLM模型测量）和OCR质量评分等字段。数据集支持按主题子集或单个来源数据集加载，适用于自然语言处理任务，特别是语言建模。

The German Commons dataset is a large-scale collection of German-language text data under open licenses, designed for training German language models. It aggregates 154.56 billion tokens from 41 diverse sources, comprising 35.78 million documents across 7 thematic domains: Web Commons, Political Commons, Legal Commons, News Commons, Economic Commons, Cultural Commons, and Scientific Commons. Each record includes fields such as a unique identifier, source dataset name, thematic subset, main text content, list of applicable licenses, GPT-2 token count, text perplexity (measured with a KenLM model), and OCR quality score. The dataset supports loading by thematic subset or individual source datasets and is suitable for natural language processing tasks, particularly language modeling.