langdonholmes/slimpajama-ngrams

---
task_categories:
- feature-extraction
size_categories:
- 100M<n<1B
---
# Dataset Card for slimpajama-ngrams

N-gram frequencies calculated from the SlimPajama test set. Github and StackExchange data sources were excluded to avoid overrepresentation of code.

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

#### Processing
N-grams were extracted for all words and for content words (cw) only.
Content words were determined by the SpaCy large pipeline version 3.5. Verbs operating as auxiliaries were treated as function words.

#### Format
Document boundaries are indicated by three "#" tokens.
Data is represented in a 4-gram format, but smaller n-grams can be recovered.

Example: the last bigram of a document will be [word_a, word_b, "#", "#"]

- **Curated by:** Langdon
- **License:**

Please refer to the licenses of the SlimPajama data subsets.

- [Common Crawl Foundation Terms of Use](https://commoncrawl.org/terms-of-use/full/)
- [C4 license](https://huggingface.co/datasets/allenai/c4#license)
- Books: [the_pile_books3 license](https://huggingface.co/datasets/the_pile_books3#licensing-information) and [pg19 license](https://huggingface.co/datasets/pg19#licensing-information)
- [ArXiv Terms of Use](https://info.arxiv.org/help/api/tou.html)
- [Wikipedia License](https://huggingface.co/datasets/wikipedia#licensing-information)

## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]