usvsnsp/memories-semantic-memorization-filter-results

--- dataset_info: features: - name: sequence_id dtype: int64 - name: text dtype: string - name: sequence_duplicates dtype: int64 - name: max_frequency dtype: int64 - name: avg_frequency dtype: float64 - name: min_frequency dtype: int64 - name: median_frequency dtype: float64 - name: p25_frequency dtype: int64 - name: p75_frequency dtype: int64 - name: frequencies sequence: int64 - name: is_incrementing dtype: bool - name: tokens sequence: int64 - name: repeating_offset dtype: int32 - name: num_repeating dtype: int32 - name: smallest_repeating_chunk sequence: int64 - name: memorization_score dtype: float64 - name: templating_frequency_0.9 dtype: int64 - name: templating_frequency_0.8 dtype: int64 - name: prompt_perplexity dtype: float32 - name: generation_perplexity dtype: float32 - name: sequence_perplexity dtype: float32 splits: - name: memories.duped.70m num_bytes: 648141277 num_examples: 463953 - name: memories.duped.160m num_bytes: 955903849 num_examples: 689673 - name: memories.duped.410m num_bytes: 1337555782 num_examples: 970341 - name: memories.duped.1b num_bytes: 1725540452 num_examples: 1256141 - name: memories.duped.1.4b num_bytes: 1884519155 num_examples: 1373722 - name: memories.duped.2.8b num_bytes: 2292743123 num_examples: 1675077 - name: memories.duped.6.9b num_bytes: 2898035658 num_examples: 2120976 - name: memories.duped.12b num_bytes: 3252649684 num_examples: 2382328 - name: memories.deduped.70m num_bytes: 576211560 num_examples: 411448 - name: memories.deduped.160m num_bytes: 809545073 num_examples: 581195 - name: memories.deduped.410m num_bytes: 1126006111 num_examples: 811039 - name: memories.deduped.1b num_bytes: 1430399436 num_examples: 1032865 - name: memories.deduped.1.4b num_bytes: 1450336662 num_examples: 1048097 - name: memories.deduped.2.8b num_bytes: 1871907415 num_examples: 1355211 - name: memories.deduped.6.9b num_bytes: 2319039796 num_examples: 1680294 - name: memories.deduped.12b num_bytes: 2581349436 num_examples: 1871216 download_size: 9223426756 dataset_size: 27159884469 --- # Dataset Card for "memories-semantic-memorization-filter-results" [More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

原始信息汇总

数据集概述

特征信息

• sequence_id: 数据类型为 int64
• text: 数据类型为 string
• sequence_duplicates: 数据类型为 int64
• max_frequency: 数据类型为 int64
• avg_frequency: 数据类型为 float64
• min_frequency: 数据类型为 int64
• median_frequency: 数据类型为 float64
• p25_frequency: 数据类型为 int64
• p75_frequency: 数据类型为 int64
• frequencies: 数据类型为 int64 序列
• is_incrementing: 数据类型为 bool
• tokens: 数据类型为 int64 序列
• repeating_offset: 数据类型为 int32
• num_repeating: 数据类型为 int32
• smallest_repeating_chunk: 数据类型为 int64 序列
• memorization_score: 数据类型为 float64
• templating_frequency_0.9: 数据类型为 int64
• templating_frequency_0.8: 数据类型为 int64
• prompt_perplexity: 数据类型为 float32
• generation_perplexity: 数据类型为 float32
• sequence_perplexity: 数据类型为 float32

数据集分割

• memories.duped.70m: 字节数为 648141277，样本数为 463953
• memories.duped.160m: 字节数为 955903849，样本数为 689673
• memories.duped.410m: 字节数为 1337555782，样本数为 970341
• memories.duped.1b: 字节数为 1725540452，样本数为 1256141
• memories.duped.1.4b: 字节数为 1884519155，样本数为 1373722
• memories.duped.2.8b: 字节数为 2292743123，样本数为 1675077
• memories.duped.6.9b: 字节数为 2898035658，样本数为 2120976
• memories.duped.12b: 字节数为 3252649684，样本数为 2382328
• memories.deduped.70m: 字节数为 576211560，样本数为 411448
• memories.deduped.160m: 字节数为 809545073，样本数为 581195
• memories.deduped.410m: 字节数为 1126006111，样本数为 811039
• memories.deduped.1b: 字节数为 1430399436，样本数为 1032865
• memories.deduped.1.4b: 字节数为 1450336662，样本数为 1048097
• memories.deduped.2.8b: 字节数为 1871907415，样本数为 1355211
• memories.deduped.6.9b: 字节数为 2319039796，样本数为 1680294
• memories.deduped.12b: 字节数为 2581349436，样本数为 1871216

数据集大小

• 下载大小: 9223426756 字节
• 数据集大小: 27159884469 字节