LisaMegaWatts/bookcorpus-gutenberg-classics

---
license: apache-2.0
language:
- en
task_categories:
- text-generation
tags:
- bookcorpus
- gutenberg
- project-gutenberg
- philosophy
- classical-texts
- character-level
- curriculum-learning
- slm
size_categories:
- 10M<n<100M
dataset_info:
features:
- name: text
dtype: string
splits:
- name: train
num_bytes: 2128762265
num_examples: 13556975
- name: validation
num_bytes: 236146830
num_examples: 1506331
download_size: 1610933332
dataset_size: 2364909095
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: validation
path: data/validation-*
---

# BookCorpus + Gutenberg Classics Training Corpus

Large-scale training corpus combining BookCorpus fiction, Project Gutenberg 19th-century literature (PG-19), and curated classical philosophy texts. Cleaned, deduplicated, and organized into curriculum phases for character-level language model training.

## Dataset Description

This corpus is the primary training dataset for the Julia SLM project, combining three major text sources into a unified, cleaned training set with curriculum-phase annotations for structured learning.

### Source Composition

| Source | Files | Chunks (pre-dedup) | Proportion |
|--------|------:|-------------------:|-----------:|
| [BookCorpus](https://huggingface.co/datasets/bookcorpus/bookcorpus) | 147 | 14,190,796 | 89.4% |
| [PG-19](https://huggingface.co/datasets/deepmind/pg19) (Project Gutenberg) | 552 | 1,344,777 | 8.5% |
| Classical Philosophy (MIT Classics, Internet Archive, Gutenberg) | 137 | 330,954 | 2.1% |
| **Total** | **836** | **15,866,527** | **100%** |

### Cleaning Applied

All text has been processed through a multi-stage cleaning pipeline:

- **Character filtering**: Lowercased to ASCII set `a-z .,;:?!'"()-`
- **Source-specific cleaning**:
- BookCorpus: Moses-style detokenization (recombined subword artifacts)
- PG-19: Gutenberg boilerplate header/footer removal
- Philosophy: LaTeX artifact removal, footnote/reference stripping
- **Deduplication**: Exact dedup removed 803,221 duplicates (5.1%)
- **Whitespace normalization**: Multi-space collapse, empty line removal

### Curriculum Phases

The corpus is organized into three curriculum phases based on the classical trivium/quadrivium education model, suitable for DoReMi-style weighted phase sampling:

| Phase | Description | Train Chunks | Proportion |
|-------|-------------|-------------:|-----------:|
| **Trivium** | Grammar, rhetoric, logic (BookCorpus fiction, classical literature, rhetoric) | 13,475,278 | 99.4% |
| **Quadrivium** | Arithmetic, geometry, music, astronomy (Aristotle Physics, Plato Timaeus, Euclid) | 11,652 | 0.08% |
| **Philosophy** | Pure philosophy (Kant, Spinoza, Bacon, Seneca, Schopenhauer) | 70,042 | 0.52% |

Phase-specific training files are available in the `curriculum/` directory.

### Dataset Statistics

| Split | Examples | Size |
|-------|----------|------|
| Train | 13,556,974 | 2.0 GB |
| Validation | 1,506,330 | 221 MB |

- **90/10 train/validation split** (shuffled)
- **Weighted phase sampling** applied per config: trivium 40%, quadrivium 35%, philosophy 25%

### Philosophy Sources

The corpus includes texts from 50+ classical authors spanning Greek, Roman, Medieval, Enlightenment, and Modern philosophy:

**Greek**: Aristotle (Metaphysics, Nicomachean Ethics, Politics, Physics, Rhetoric, Poetics, Categories, Prior/Posterior Analytics, Topics, On the Soul, On the Heavens, On Interpretation, Generation and Corruption), Plato (Republic, Laws, Timaeus, Phaedo, Phaedrus, Symposium, Meno, Theaetetus, Protagoras), Herodotus, Thucydides, Xenophon, Aeschylus, Sophocles, Homer, Euripides

**Roman**: Marcus Aurelius, Seneca, Epictetus, Cicero, Lucretius, Plutarch, Tacitus, Virgil

**Medieval/Renaissance**: Boethius, Machiavelli, Thomas More

**Enlightenment**: Descartes, Spinoza, Leibniz, Locke, Berkeley, Hume, Kant, Rousseau, Montesquieu

**Modern**: Schopenhauer, Mill, Thoreau, William James

**Eastern**: Bhagavad Gita, Sun Tzu, Confucius, Lao Tzu

## Additional Files

The `curriculum/` directory contains phase-specific training files:
- `train_trivium.txt` - Grammar, rhetoric, and logic texts (2.0 GB)
- `train_quadrivium.txt` - Mathematical and natural philosophy texts (2.1 MB)
- `train_philosophy.txt` - Pure philosophy texts (13 MB)

## Usage

```python
from datasets import load_dataset

ds = load_dataset("LisaMegaWatts/bookcorpus-gutenberg-classics")

# Training data
for example in ds["train"]:
text = example["text"]

# Download phase-specific files for curriculum training
from huggingface_hub import hf_hub_download

trivium = hf_hub_download(
"LisaMegaWatts/bookcorpus-gutenberg-classics",
"curriculum/train_trivium.txt",
repo_type="dataset",
)
```

## Related Datasets

- [LisaMegaWatts/philosophy-corpus](https://huggingface.co/datasets/LisaMegaWatts/philosophy-corpus) - Isolated philosophy provenance dataset (for model training lineage)
- [LisaMegaWatts/wikitext-103-quality-scored](https://huggingface.co/datasets/LisaMegaWatts/wikitext-103-quality-scored) - Quality-scored WikiText-103 (Wikipedia Featured articles)
- [LisaMegaWatts/classical-humanities-corpus](https://huggingface.co/datasets/LisaMegaWatts/classical-humanities-corpus) - Extended classical humanities collection

## License

Apache 2.0

许可证：Apache-2.0
语言：英语
任务类别：文本生成
标签：BookCorpus语料库（BookCorpus）、古腾堡（Gutenberg）、古腾堡计划（Project Gutenberg）、哲学、古典文本、字符级、课程学习、小语言模型（Small Language Model，SLM）
规模类别：10M<n<100M
数据集信息：
特征：
- 名称：text，数据类型：字符串
划分集：
- 训练集（train）：字节数2128762265，样本数13556975
- 验证集（validation）：字节数236146830，样本数1506331
下载大小：1610933332，数据集总大小：2364909095
配置：
- 配置名称：default，数据文件：
- 训练集：路径data/train-*
- 验证集：路径data/validation-*

# BookCorpus与古腾堡经典文本训练语料库

该大规模训练语料库整合了BookCorpus小说文本、古腾堡计划19世纪文学语料库（PG-19）以及精选古典哲学文本，经过清洗、去重处理，并按课程学习阶段组织，适配字符级语言模型训练。

## 数据集说明
该语料库是Julia小语言模型（Small Language Model，SLM）项目的核心训练数据集，整合三大文本来源，构建了统一的清洗后训练集，并添加课程学习阶段标注以支持结构化训练。

### 数据源构成
| 数据源 | 文件数 | 去重前块数 | 占比 |
|--------|------:|-------------------:|-----------:|
| BookCorpus语料库（BookCorpus） | 147 | 14,190,796 | 89.4% |
| PG-19（古腾堡计划） | 552 | 1,344,777 | 8.5% |
| 古典哲学文本（MIT古典文献库、互联网档案馆、古腾堡计划） | 137 | 330,954 | 2.1% |
| **总计** | **836** | **15,866,527** | **100%** |

### 预处理清洗流程
所有文本均经过多阶段清洗流水线处理：
- **字符过滤**：统一转为小写ASCII字符集`a-z .,;:?!'"()-`
- **源特定清洗**：
- BookCorpus语料库：采用摩西风格（Moses-style）去分词处理，重组子词伪影
- PG-19：移除古腾堡计划文本的模板化页眉与页脚
- 哲学文本：移除LaTeX伪影，剥离脚注与参考文献
- **去重**：通过精确去重移除了803,221条重复样本，占比5.1%
- **空格标准化**：合并多空格，移除空行

### 课程学习阶段
该语料库基于古典七艺中的三艺/四艺教育模型划分为三个课程阶段，适配DoReMi风格的加权阶段采样：
| 阶段 | 描述 | 训练块数 | 占比 |
|-------|-------------|-------------:|-----------:|
| **三艺（Trivium）** | 语法、修辞、逻辑（BookCorpus小说文本、古典文学、修辞学文本） | 13,475,278 | 99.4% |
| **四艺（Quadrivium）** | 算术、几何、音乐、天文（亚里士多德《物理学》、柏拉图《蒂迈欧篇》、欧几里得著作） | 11,652 | 0.08% |
| **哲学（Philosophy）** | 纯哲学文本（康德、斯宾诺莎、培根、塞涅卡、叔本华著作） | 70,042 | 0.52% |

各阶段专属的训练文件可在`curriculum/`目录中获取。

### 数据集统计信息
| 划分集 | 样本数 | 大小 |
|-------|----------|------|
| 训练集 | 13,556,974 | 2.0 GB |
| 验证集 | 1,506,330 | 221 MB |

- **训练/验证划分比例为90/10**（已打乱）
- **各配置采用加权阶段采样**：三艺40%、四艺35%、哲学25%

### 哲学文本数据源
该语料库收录了50余位古典作家的文本，涵盖古希腊、古罗马、中世纪、启蒙运动与近现代哲学领域：
**古希腊**：亚里士多德（《形而上学》《尼各马可伦理学》《政治学》《物理学》《修辞学》《诗学》《范畴篇》《前/后分析篇》《论题篇》《论灵魂》《论天》《解释篇》《论生成与毁灭》）、柏拉图（《理想国》《法篇》《蒂迈欧篇》《斐多篇》《斐德罗篇》《会饮篇》《美诺篇》《泰阿泰德篇》《普罗泰戈拉篇》）、希罗多德、修昔底德、色诺芬、埃斯库罗斯、索福克勒斯、荷马、欧里庇得斯
**古罗马**：马可·奥勒留、塞涅卡、爱比克泰德、西塞罗、卢克莱修、普鲁塔克、塔西佗、维吉尔
**中世纪/文艺复兴**：波爱修斯、马基雅维利、托马斯·莫尔
**启蒙运动**：笛卡尔、斯宾诺莎、莱布尼茨、洛克、贝克莱、休谟、康德、卢梭、孟德斯鸠
**近现代**：叔本华、密尔、梭罗、威廉·詹姆斯
**东方哲学**：《薄伽梵歌》、孙子、孔子、老子

## 附加文件
`curriculum/`目录包含各阶段专属训练文件：
- `train_trivium.txt`：语法、修辞与逻辑文本（2.0 GB）
- `train_quadrivium.txt`：数学与自然哲学文本（2.1 MB）
- `train_philosophy.txt`：纯哲学文本（13 MB）

## 使用示例
python
from datasets import load_dataset

ds = load_dataset("LisaMegaWatts/bookcorpus-gutenberg-classics")

# 遍历训练集样本
for example in ds["train"]:
text = example["text"]

# 下载阶段专属训练文件以用于课程学习训练
from huggingface_hub import hf_hub_download

trivium = hf_hub_download(
"LisaMegaWatts/bookcorpus-gutenberg-classics",
"curriculum/train_trivium.txt",
repo_type="dataset",
)

## 相关数据集
- [LisaMegaWatts/philosophy-corpus](https://huggingface.co/datasets/LisaMegaWatts/philosophy-corpus) - 独立哲学来源语料库（用于模型训练溯源）
- [LisaMegaWatts/wikitext-103-quality-scored](https://huggingface.co/datasets/LisaMegaWatts/wikitext-103-quality-scored) - 带质量评分的WikiText-103（维基百科精选文章）
- [LisaMegaWatts/classical-humanities-corpus](https://huggingface.co/datasets/LisaMegaWatts/classical-humanities-corpus) - 扩展古典人文社科语料库

## 许可证
Apache 2.0