pszemraj/LoC-PD-Books-preprocessed

---
language:
- en
license: cc0-1.0
size_categories:
- 1K<n<10K
source_datasets: storytracer/LoC-PD-Books
task_categories:
- text-generation
- feature-extraction
dataset_info:
- config_name: default
features:
- name: lccn
dtype: string
- name: title
dtype: string
- name: author
dtype: string
- name: year
dtype: int64
- name: page_count
dtype: int64
- name: filename
dtype: string
- name: text
dtype: string
- name: label
dtype: string
- name: score
dtype: float64
splits:
- name: train
num_bytes: 2788098633.628336
num_examples: 8816
download_size: 1435586557
dataset_size: 2788098633.628336
- config_name: en-clean
features:
- name: lccn
dtype: string
- name: title
dtype: string
- name: author
dtype: string
- name: year
dtype: int64
- name: page_count
dtype: int64
- name: filename
dtype: string
- name: text
dtype: string
- name: score
dtype: float64
splits:
- name: train
num_bytes: 1906155961.9587114
num_examples: 6399
download_size: 1055862380
dataset_size: 1906155961.9587114
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- config_name: en-clean
data_files:
- split: train
path: en-clean/train-*
tags:
- books
---

# LoC-PD-Books: preprocessed

This is the `storytracer/LoC-PD-Books` dataset with the following preprocessing steps:

- apply [clean-text](https://pypi.org/project/clean-text/) package keeping casing and newlines
- drop OCR garbled text in first few lines of each example
- fix (most) 'hard' newlines w/ regex similar to [gutenberg clean](https://huggingface.co/datasets/BEE-spoke-data/gutenberg-en-v1-clean)
- 'grade' first 512 tokens of each book with [this quantized model](https://huggingface.co/pszemraj/gibberish_detector_onnx-quant-avx512_vnni); keep examples from labels `clean` (all) and `mild gibberish` w/ score 0.9 or higher

---

原始信息汇总

数据集概述

基本信息

• 语言: 英语
• 许可: CC0-1.0
• 大小分类: 1K<n<10K
• 源数据集: storytracer/LoC-PD-Books
• 任务类别:
  • 文本生成
  • 特征提取

数据集配置

默认配置 (config_name: default)

• 特征:
  • lccn: 字符串
  • title: 字符串
  • author: 字符串
  • year: 整数
  • page_count: 整数
  • filename: 字符串
  • text: 字符串
  • label: 字符串
  • score: 浮点数
• 分割:
  • 训练集:
    • 字节数: 2788098633.628336
    • 示例数: 8816
  • 下载大小: 1435586557
  • 数据集大小: 2788098633.628336

en-clean配置 (config_name: en-clean)

• 特征:
  • lccn: 字符串
  • title: 字符串
  • author: 字符串
  • year: 整数
  • page_count: 整数
  • filename: 字符串
  • text: 字符串
  • score: 浮点数
• 分割:
  • 训练集:
    • 字节数: 1906155961.9587114
    • 示例数: 6399
  • 下载大小: 1055862380
  • 数据集大小: 1906155961.9587114

数据文件

• 默认配置:
  • 训练集路径: data/train-*
• en-clean配置:
  • 训练集路径: en-clean/train-*

标签

• 书籍