text-machine-lab/unconstrained_language

---
dataset_info:
features:
- name: TEXT
dtype: string
splits:
- name: train
num_bytes: 5437652389
num_examples: 9081490
- name: validation
num_bytes: 50107745
num_examples: 100000
- name: test
num_bytes: 50134861
num_examples: 100000
download_size: 3732550490
dataset_size: 5537894995
---

# Dataset Card for unconstrained_language

## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Citation Information](#additional-information)
- [Citation Information](#citation-information)

## Dataset Description

- **Paper: https://arxiv.org/abs/2305.17266**
- **Point of Contact: [e-mail]**

### Dataset Summary

This dataset is one of the two datasets published by "Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale" (https://arxiv.org/abs/2305.17266).
The dataset available at this link is the pre-training data **not** constrained by any predefined vocabulary. The other published data i.e. the pre-training data that is constrained by vocabulary is available at https://huggingface.co/datasets/text-machine-lab/constrained_language.

This dataset is curated by randomly sampling text spans (of an approximate length of 128 tokens) from the following corpora,
- C4: https://arxiv.org/abs/1910.10683,
- BookCorpus: https://ieeexplore.ieee.org/document/7410368,
- Wikipedia: https://huggingface.co/datasets/wikipedia,
- Simplified-Wikipedia: https://simple.wikipedia.org/wiki/Main_Page,
- Children's Book Test Corpus: https://arxiv.org/abs/1511.02301

The dataset includes ~9 million contiguous spans, each with approximately 128 tokens.

### Languages

The dataset contains the English language only.

## Dataset Structure

The dataset is available in the arrow dataset format with three splits namely, train, validation, and test. Every data instance has only one key "Text" that included a text span of approximately 128 tokens.

### Citation Information
If this dataset is useful to you please cite our work.

```sh
@article{deshpande2023honey,
title={Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale},
author={Deshpande, Vijeta and Pechi, Dan and Thatte, Shree and Lialin, Vladislav and Rumshisky, Anna},
journal={arXiv preprint arXiv:2305.17266},
year={2023}
}
```

原始信息汇总

数据集概述

数据集名称

• 名称: unconstrained_language

数据集特征

• 特征名称: TEXT
• 数据类型: string

数据集结构

• 数据分割:
  • 训练集:
    • 示例数量: 9081490
    • 数据大小: 5437652389 bytes
  • 验证集:
    • 示例数量: 100000
    • 数据大小: 50107745 bytes
  • 测试集:
    • 示例数量: 100000
    • 数据大小: 50134861 bytes

数据集大小

• 下载大小: 3732550490 bytes
• 总数据集大小: 5537894995 bytes

数据集来源

• 数据来源:
  • C4
  • BookCorpus
  • Wikipedia
  • Simplified-Wikipedia
  • Childrens Book Test Corpus
• 数据类型: 随机抽样的文本片段，每个片段约128个tokens

语言

• 语言: 英语

引用信息

• 引用格式: sh @article{deshpande2023honey, title={Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale}, author={Deshpande, Vijeta and Pechi, Dan and Thatte, Shree and Lialin, Vladislav and Rumshisky, Anna}, journal={arXiv preprint arXiv:2305.17266}, year={2023} }