usvsnsp/pile-semantic-memorization-filter-results

--- dataset_info: features: - name: sequence_id dtype: int64 - name: text dtype: string - name: sequence_duplicates dtype: int64 - name: max_frequency dtype: int64 - name: avg_frequency dtype: float64 - name: min_frequency dtype: int64 - name: median_frequency dtype: float64 - name: p25_frequency dtype: int64 - name: p75_frequency dtype: int64 - name: frequencies sequence: int64 - name: is_incrementing dtype: bool - name: tokens sequence: int64 - name: repeating_offset dtype: int32 - name: num_repeating dtype: int32 - name: smallest_repeating_chunk sequence: int64 - name: memorization_score dtype: float64 - name: templating_frequency_0.9 dtype: int64 - name: templating_frequency_0.8 dtype: int64 - name: prompt_perplexity dtype: float32 - name: generation_perplexity dtype: float32 - name: sequence_perplexity dtype: float32 splits: - name: pile.duped.70m num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.160m num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.410m num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.1b num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.1.4b num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.2.8b num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.6.9b num_bytes: 7003348430 num_examples: 5000000 - name: pile.duped.12b num_bytes: 7003348430 num_examples: 5000000 - name: pile.deduped.70m num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.160m num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.410m num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.1b num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.1.4b num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.2.8b num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.6.9b num_bytes: 7013409756 num_examples: 5000000 - name: pile.deduped.12b num_bytes: 7013409756 num_examples: 5000000 download_size: 48107269588 dataset_size: 112134065488 --- # Dataset Card for "pile-semantic-memorization-filter-results" [More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

原始信息汇总

数据集概述

特征信息

• sequence_id: 数据类型为 int64
• text: 数据类型为 string
• sequence_duplicates: 数据类型为 int64
• max_frequency: 数据类型为 int64
• avg_frequency: 数据类型为 float64
• min_frequency: 数据类型为 int64
• median_frequency: 数据类型为 float64
• p25_frequency: 数据类型为 int64
• p75_frequency: 数据类型为 int64
• frequencies: 数据类型为 int64 的序列
• is_incrementing: 数据类型为 bool
• tokens: 数据类型为 int64 的序列
• repeating_offset: 数据类型为 int32
• num_repeating: 数据类型为 int32
• smallest_repeating_chunk: 数据类型为 int64 的序列
• memorization_score: 数据类型为 float64
• templating_frequency_0.9: 数据类型为 int64
• templating_frequency_0.8: 数据类型为 int64
• prompt_perplexity: 数据类型为 float32
• generation_perplexity: 数据类型为 float32
• sequence_perplexity: 数据类型为 float32

数据分割

• pile.duped.70m: 字节数为 7003348430，样本数为 5000000
• pile.duped.160m: 字节数为 7003348430，样本数为 5000000
• pile.duped.410m: 字节数为 7003348430，样本数为 5000000
• pile.duped.1b: 字节数为 7003348430，样本数为 5000000
• pile.duped.1.4b: 字节数为 7003348430，样本数为 5000000
• pile.duped.2.8b: 字节数为 7003348430，样本数为 5000000
• pile.duped.6.9b: 字节数为 7003348430，样本数为 5000000
• pile.duped.12b: 字节数为 7003348430，样本数为 5000000
• pile.deduped.70m: 字节数为 7013409756，样本数为 5000000
• pile.deduped.160m: 字节数为 7013409756，样本数为 5000000
• pile.deduped.410m: 字节数为 7013409756，样本数为 5000000
• pile.deduped.1b: 字节数为 7013409756，样本数为 5000000
• pile.deduped.1.4b: 字节数为 7013409756，样本数为 5000000
• pile.deduped.2.8b: 字节数为 7013409756，样本数为 5000000
• pile.deduped.6.9b: 字节数为 7013409756，样本数为 5000000
• pile.deduped.12b: 字节数为 7013409756，样本数为 5000000

数据集大小

• 下载大小: 48107269588 字节
• 数据集大小: 112134065488 字节