recursal/LectureGratuits

---
annotations_creators:
- no-annotation
language_creators:
- crowdsourced
license: cc-by-sa-4.0
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
- masked-language-modeling
source_datasets:
- original
language:
- en
pretty_name: Lecture Gratuits

configs:
- config_name: default
data_files:
- split: final
path: data/*
---

# Dataset Card for LectureGratuits


*Waifu to catch your attention.*

## Dataset Details

### Dataset Description

*LectureGratuits* is a cleaned dataset of [*Ebooks Gratuits*](https://www.ebooksgratuits.com/) books. We downloaded all the publicly available ebooks books at the time and processed them.
Filtering to a total amount of tokens of **~265.46M** (llama-2-7b-chat-tokenizer) / **~253.51M** (RWKV Tokenizer) from primarily English language.

- **Curated by:** Darok
- **Funded by:** Recursal.ai
- **Shared by:** KaraKaraWitch
- **Language(s) (NLP):** English
- **License:** Public domain

### Dataset Sources

- **Source Data:** [ebooksgratuits.com](https://www.ebooksgratuits.com)

### Processing

KaraKaraWitch doesn't have specifics on how it's processed. We have postiluated the following workflow / processing:

0. Get the higher ID
1. Enumerate and download all the epub files: `https://www.ebooksgratuits.com/newsendbook.php?id=<ID>&format=epub`
2. Put them in a folder called `books`
3. extract content to each json file in `output` folder. (See filtering steps in `extract-text.py`)
4. Combine into a single file.

### Data Keys

```
text (str): The book's text. Converted to markdown.
meta (dict): A dictionary of metadata with the following keys:
- title
- author
- publisher
```

### Dataset Curators

This dataset was mainly Darok's work. I (KaraKaraWitch) only assisted them with questions and the writing of the dataset card.

### Licensing Information

The books itself is in public domain. For the post processed data under Recursal work, it's licensed as CC-BY-SA.

Recursal Waifus (The banner image) are licensed under CC-BY-SA.
They do not represent the related websites in any official capacity unless otherwise or announced by the website.
You may use them as a banner image. However, you must always link back to the dataset.

### Citation Information

```
@ONLINE{lecturegratuits,
title = {LectureGratuits},
author = {Darok, KaraKaraWitch, recursal.ai},
year = {2024},
howpublished = {\url{https://huggingface.co/datasets/recursal/Recursalberg}},
}
```

原始信息汇总

LectureGratuits 数据集概述

数据集描述

• 名称: LectureGratuits
• 来源: ebooksgratuits.com
• 语言: 英语
• 总词数: 约265.46M（llama-2-7b-chat-tokenizer）/ 约253.51M（RWKV Tokenizer）
• 许可证: 公共领域
• 处理后数据许可证: CC-BY-SA

数据集来源

• 原始数据: ebooksgratuits.com

数据处理

1. 获取最高ID
2. 枚举并下载所有epub文件: https://www.ebooksgratuits.com/newsendbook.php?id=<ID>&format=epub
3. 将文件放入名为books的文件夹
4. 提取内容到output文件夹中的每个json文件（参见extract-text.py中的过滤步骤）
5. 合并成单个文件

数据键

• text (str): 书籍文本，转换为markdown格式
• meta (dict): 包含以下键的元数据字典:
  • title
  • author
  • publisher

数据集贡献者

• 主要贡献者: Darok
• 协助者: KaraKaraWitch

许可证信息

• 书籍: 公共领域
• 处理后数据: CC-BY-SA

引用信息

@ONLINE{lecturegratuits, title = {LectureGratuits}, author = {Darok, KaraKaraWitch, recursal.ai}, year = {2024}, howpublished = {url{https://huggingface.co/datasets/recursal/Recursalberg}}, }