CarperAI/pile-v2-small-filtered

---
annotations_creators: []
language_creators:
- crowdsourced
language: ["en","code"]
multilinguality:
- multilingual
size_categories:
- unknown
source_datasets: []
task_categories:
- text-generation
task_ids:
- language-modeling

---

## Dataset Description

A small subset in each dataset of `pile-v2`(~1000 samples) of [pile-v2]() dataset, each has 1,000 random samples from the original dataset. The dataset has 255MB of text (code and english).

## Languages
The dataset contains technical text on programming languages and natural language with the following subsets,
- Bible
- TED2020
- PileOfLaw
- StackExchange
- GithubIssues
- Opensubtitles
- USPTO
- S2ORC
- DevDocs
- CodePileReddit2022
- USENET
- GNOME
- ASFPublicMail
- PileV2Reddit2020
- CodePilePosts
- Discourse
- Tanzil
- arXiv
- UbuntuIRC
- PubMed
- CodePileReddit2020
- CodePileReddit2021
- GlobalVoices
- FreeLaw_Options
- PileV2Posts

## Dataset Structure

```python
from datasets import load_dataset
load_dataset("CarperAI/pile-v2-small")
```
### How to use it
You can either load the whole dataset like above, or load a specific subset such as arxiv by specifying the folder directory:
```python
load_dataset("CarperAI/pile-v2-small", data_dir="data/arxiv")
```

原始信息汇总

数据集概述

基本信息

• 数据集名称: pile-v2-small
• 数据集大小: 255MB
• 数据类型: 文本（代码和英语）
• 样本数量: 约1000个样本
• 任务类型: 文本生成
• 任务ID: 语言建模

语言和多语言性

• 语言: 英语和代码
• 多语言性: 多语言

数据集结构

• 子集: 包括Bible, TED2020, PileOfLaw, StackExchange, GithubIssues, Opensubtitles, USPTO, S2ORC, DevDocs, CodePileReddit2022, USENET, GNOME, ASFPublicMail, PileV2Reddit2020, CodePilePosts, Discourse, Tanzil, arXiv, UbuntuIRC, PubMed, CodePileReddit2020, CodePileReddit2021, GlobalVoices, FreeLaw_Options, PileV2Posts等。

使用方法

• 加载整个数据集: python from datasets import load_dataset load_dataset("CarperAI/pile-v2-small")

• 加载特定子集: python load_dataset("CarperAI/pile-v2-small", data_dir="data/arxiv")