davanstrien/blbooks-parquet-embedded

Name: davanstrien/blbooks-parquet-embedded
Creator: davanstrien
Published: 2023-07-14 14:38:08
License: 暂无描述

Hugging Face2023-07-14 更新2024-03-04 收录

下载链接：

https://hf-mirror.com/datasets/davanstrien/blbooks-parquet-embedded

下载链接

链接失效反馈

官方服务：

资源简介：

--- annotations_creators: - no-annotation language_creators: - machine-generated language: - de - en - es - fr - it - nl license: - cc0-1.0 multilinguality: - multilingual size_categories: - 100K<n<1M source_datasets: davanstrien/blbooks-parquet task_categories: - text-generation - fill-mask - other task_ids: - language-modeling - masked-language-modeling pretty_name: British Library Books tags: - embeddings dataset_info: - config_name: all features: - name: record_id dtype: string - name: date dtype: int32 - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 30394267732 num_examples: 14011953 download_size: 10486035662 dataset_size: 30394267732 - config_name: 1800s features: - name: record_id dtype: string - name: date dtype: int32 - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 30020434670 num_examples: 13781747 download_size: 10348577602 dataset_size: 30020434670 - config_name: 1700s features: - name: record_id dtype: string - name: date dtype: int32 - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 266382657 num_examples: 178224 download_size: 95137895 dataset_size: 266382657 - config_name: '1510_1699' features: - name: record_id dtype: string - name: date dtype: timestamp[s] - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 107667469 num_examples: 51982 download_size: 42320165 dataset_size: 107667469 - config_name: '1500_1899' features: - name: record_id dtype: string - name: date dtype: timestamp[s] - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 30452067039 num_examples: 14011953 download_size: 10486035662 dataset_size: 30452067039 - config_name: '1800_1899' features: - name: record_id dtype: string - name: date dtype: timestamp[s] - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 30077284377 num_examples: 13781747 download_size: 10348577602 dataset_size: 30077284377 - config_name: '1700_1799' features: - name: record_id dtype: string - name: date dtype: timestamp[s] - name: raw_date dtype: string - name: title dtype: string - name: place dtype: string - name: empty_pg dtype: bool - name: text dtype: string - name: pg dtype: int32 - name: mean_wc_ocr dtype: float32 - name: std_wc_ocr dtype: float64 - name: name dtype: string - name: all_names dtype: string - name: Publisher dtype: string - name: Country of publication 1 dtype: string - name: all Countries of publication dtype: string - name: Physical description dtype: string - name: Language_1 dtype: string - name: Language_2 dtype: string - name: Language_3 dtype: string - name: Language_4 dtype: string - name: multi_language dtype: bool splits: - name: train num_bytes: 267117831 num_examples: 178224 download_size: 95137895 dataset_size: 267117831 --- # Dataset Card for "blbooks-parquet-embedded" [More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

# 「blbooks-parquet-embedded」数据集卡片 [需补充更多信息](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ## 数据集元数据 - 注释创建者：无注释 - 语言生成方式：机器生成 - 支持语言：德语（de）、英语（en）、西班牙语（es）、法语（fr）、意大利语（it）、荷兰语（nl） - 授权协议：CC0 1.0 - 多语言属性：多语言 - 样本规模：100,000 < 样本数量 < 1,000,000 - 源数据集：`davanstrien/blbooks-parquet` - 任务类别：文本生成、掩码填充、其他 - 任务子类型：语言建模、掩码语言建模 - 易读名称：大英图书馆图书（British Library Books） - 标签：嵌入（embeddings） ## 数据集配置详情 ### 配置名：all #### 特征字段： 1. 记录ID（record_id）：字符串类型 2. 日期（date）：32位整数类型 3. 原始日期（raw_date）：字符串类型 4. 书名（title）：字符串类型 5. 出版地（place）：字符串类型 6. 是否空页（empty_pg）：布尔类型 7. 文本内容（text）：字符串类型 8. `pg`：32位整数类型 9. OCR平均词数（mean_wc_ocr）：32位浮点数类型 10. OCR词数标准差（std_wc_ocr）：64位浮点数类型 11. 名称（name）：字符串类型 12. 所有名称（all_names）：字符串类型 13. 出版社（Publisher）：字符串类型 14. 第一出版国（Country of publication 1）：字符串类型 15. 所有出版国（all Countries of publication）：字符串类型 16. 实体描述（Physical description）：字符串类型 17. 第一语言（Language_1）：字符串类型 18. 第二语言（Language_2）：字符串类型 19. 第三语言（Language_3）：字符串类型 20. 第四语言（Language_4）：字符串类型 21. 是否多语言（multi_language）：布尔类型 #### 数据划分： - 训练集（train）：字节数30394267732，样本量14011953 - 下载大小：10486035662 - 数据集总大小：30394267732 ### 配置名：1800s #### 特征字段：与配置`all`完全一致 #### 数据划分： - 训练集（train）：字节数30020434670，样本量13781747 - 下载大小：10348577602 - 数据集总大小：30020434670 ### 配置名：1700s #### 特征字段：与配置`all`完全一致 #### 数据划分： - 训练集（train）：字节数266382657，样本量178224 - 下载大小：95137895 - 数据集总大小：266382657 ### 配置名：1510_1699 #### 特征字段：仅日期（date）类型为秒级时间戳（timestamp[s]），其余字段与配置`all`一致 #### 数据划分： - 训练集（train）：字节数107667469，样本量51982 - 下载大小：42320165 - 数据集总大小：107667469 ### 配置名：1500_1899 #### 特征字段：仅日期（date）类型为秒级时间戳（timestamp[s]），其余字段与配置`all`一致 #### 数据划分： - 训练集（train）：字节数30452067039，样本量14011953 - 下载大小：10486035662 - 数据集总大小：30452067039 ### 配置名：1800_1899 #### 特征字段：仅日期（date）类型为秒级时间戳（timestamp[s]），其余字段与配置`all`一致 #### 数据划分： - 训练集（train）：字节数30077284377，样本量13781747 - 下载大小：10348577602 - 数据集总大小：30077284377 ### 配置名：1700_1799 #### 特征字段：仅日期（date）类型为秒级时间戳（timestamp[s]），其余字段与配置`all`一致 #### 数据划分： - 训练集（train）：字节数267117831，样本量178224 - 下载大小：95137895 - 数据集总大小：267117831

提供机构：

davanstrien

原始信息汇总

数据集概述

基本信息

名称: British Library Books
语言: 多语言（de, en, es, fr, it, nl）
许可证: CC0-1.0
多语言性: 多语言
大小: 100K<n<1M

来源与任务

来源数据集: davanstrien/blbooks-parquet
任务类别: text-generation, fill-mask, other
任务ID: language-modeling, masked-language-modeling

数据集配置与特征

配置名称: all, 1800s, 1700s, 1510_1699, 1500_1899, 1800_1899, 1700_1799
特征:
- record_id: string
- date: int32 或 timestamp[s]
- raw_date: string
- title: string
- place: string
- empty_pg: bool
- text: string
- pg: int32
- mean_wc_ocr: float32
- std_wc_ocr: float64
- name: string
- all_names: string
- Publisher: string
- Country of publication 1: string
- all Countries of publication: string
- Physical description: string
- Language_1, Language_2, Language_3, Language_4: string
- multi_language: bool

数据集拆分

训练集:
- all:
  - num_bytes: 30394267732
  - num_examples: 14011953
  - download_size: 10486035662
  - dataset_size: 30394267732
- 1800s:
  - num_bytes: 30020434670
  - num_examples: 13781747
  - download_size: 10348577602
  - dataset_size: 30020434670
- 1700s:
  - num_bytes: 266382657
  - num_examples: 178224
  - download_size: 95137895
  - dataset_size: 266382657
- 1510_1699:
  - num_bytes: 107667469
  - num_examples: 51982
  - download_size: 42320165
  - dataset_size: 107667469
- 1500_1899:
  - num_bytes: 30452067039
  - num_examples: 14011953
  - download_size: 10486035662
  - dataset_size: 30452067039
- 1800_1899:
  - num_bytes: 30077284377
  - num_examples: 13781747
  - download_size: 10348577602
  - dataset_size: 30077284377
- 1700_1799:
  - num_bytes: 267117831
  - num_examples: 178224
  - download_size: 95137895
  - dataset_size: 267117831