usvsnsp/pile-semantic-memorization-filter-results
收藏Hugging Face2023-09-19 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/usvsnsp/pile-semantic-memorization-filter-results
下载链接
链接失效反馈官方服务：
资源简介：
---
dataset_info:
features:
- name: sequence_id
dtype: int64
- name: text
dtype: string
- name: sequence_duplicates
dtype: int64
- name: max_frequency
dtype: int64
- name: avg_frequency
dtype: float64
- name: min_frequency
dtype: int64
- name: median_frequency
dtype: float64
- name: p25_frequency
dtype: int64
- name: p75_frequency
dtype: int64
- name: frequencies
sequence: int64
- name: is_incrementing
dtype: bool
- name: tokens
sequence: int64
- name: repeating_offset
dtype: int32
- name: num_repeating
dtype: int32
- name: smallest_repeating_chunk
sequence: int64
- name: memorization_score
dtype: float64
- name: templating_frequency_0.9
dtype: int64
- name: templating_frequency_0.8
dtype: int64
- name: prompt_perplexity
dtype: float32
- name: generation_perplexity
dtype: float32
- name: sequence_perplexity
dtype: float32
splits:
- name: pile.duped.70m
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.160m
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.410m
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.1b
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.1.4b
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.2.8b
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.6.9b
num_bytes: 7003348430
num_examples: 5000000
- name: pile.duped.12b
num_bytes: 7003348430
num_examples: 5000000
- name: pile.deduped.70m
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.160m
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.410m
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.1b
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.1.4b
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.2.8b
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.6.9b
num_bytes: 7013409756
num_examples: 5000000
- name: pile.deduped.12b
num_bytes: 7013409756
num_examples: 5000000
download_size: 48107269588
dataset_size: 112134065488
---
# Dataset Card for "pile-semantic-memorization-filter-results"
[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
提供机构：
usvsnsp
原始信息汇总
数据集概述
特征信息
- sequence_id: 数据类型为
int64 - text: 数据类型为
string - sequence_duplicates: 数据类型为
int64 - max_frequency: 数据类型为
int64 - avg_frequency: 数据类型为
float64 - min_frequency: 数据类型为
int64 - median_frequency: 数据类型为
float64 - p25_frequency: 数据类型为
int64 - p75_frequency: 数据类型为
int64 - frequencies: 数据类型为
int64的序列 - is_incrementing: 数据类型为
bool - tokens: 数据类型为
int64的序列 - repeating_offset: 数据类型为
int32 - num_repeating: 数据类型为
int32 - smallest_repeating_chunk: 数据类型为
int64的序列 - memorization_score: 数据类型为
float64 - templating_frequency_0.9: 数据类型为
int64 - templating_frequency_0.8: 数据类型为
int64 - prompt_perplexity: 数据类型为
float32 - generation_perplexity: 数据类型为
float32 - sequence_perplexity: 数据类型为
float32
数据分割
- pile.duped.70m: 字节数为 7003348430，样本数为 5000000
- pile.duped.160m: 字节数为 7003348430，样本数为 5000000
- pile.duped.410m: 字节数为 7003348430，样本数为 5000000
- pile.duped.1b: 字节数为 7003348430，样本数为 5000000
- pile.duped.1.4b: 字节数为 7003348430，样本数为 5000000
- pile.duped.2.8b: 字节数为 7003348430，样本数为 5000000
- pile.duped.6.9b: 字节数为 7003348430，样本数为 5000000
- pile.duped.12b: 字节数为 7003348430，样本数为 5000000
- pile.deduped.70m: 字节数为 7013409756，样本数为 5000000
- pile.deduped.160m: 字节数为 7013409756，样本数为 5000000
- pile.deduped.410m: 字节数为 7013409756，样本数为 5000000
- pile.deduped.1b: 字节数为 7013409756，样本数为 5000000
- pile.deduped.1.4b: 字节数为 7013409756，样本数为 5000000
- pile.deduped.2.8b: 字节数为 7013409756，样本数为 5000000
- pile.deduped.6.9b: 字节数为 7013409756，样本数为 5000000
- pile.deduped.12b: 字节数为 7013409756，样本数为 5000000
数据集大小
- 下载大小: 48107269588 字节
- 数据集大小: 112134065488 字节
