carolina-c4ai/carol-subcorpora

Subcorpora Carolina包含两个经过平衡和去重的子集Carol·B和Carol·(D+B)，它们来自Carolina Corpus (Bea)版本。Carol·B在Carolina Corpus的最大领域中每个领域的令牌数进行了平衡，大约有6020万个令牌。Carol·(D+B)旨在在令牌平衡之前减少每个领域中的重复内容，使用了Onion工具进行去重。这两个子集分别包含3亿6107万1千1百47个令牌和5.5GB，以及3亿6116万7千2百65个令牌和5.7GB。虚拟领域是最小的领域，因此在两个子集中都以完整形式呈现。

Subcorpora Carolina contains two balanced and deduplicated subsets Carol·B and Carol·(D+B) from the Carolina Corpus (Bea) version. Carol·B is balanced in terms of tokens per domain from the largest domains of Carolina Corpus, with approximately 60.2M tokens. Carol·(D+B) is aimed at reducing duplicated content from each domain before balancing tokens, using the Onion corpus tool for deduplication. These subsets contain 361,071,147 tokens and 5.5GB, and 361,167,265 tokens and 5.7GB respectively. The Virtual domain, being the smallest, is reproduced in its entirety in both subsets.