TinyStories

Dataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary.

Described in the following paper: https://arxiv.org/abs/2305.07759.

The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss). These models can be found on Huggingface, at roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M.

Additional resources:
tinystories_all_data.tar.gz - contains a superset of the stories together with metadata and the prompt that was used to create each story.

TinyStoriesV2-GPT4-train.txt - Is a new version of the dataset that is based on generations by GPT-4 only (the original dataset also has generations by GPT-3.5 which are of lesser quality). It contains all the examples in TinyStories.txt which were GPT-4 generated as a subset (but is significantly larger).

Evaluation_prompts.yaml: List of prompts used to evaluate our models (see paper)

本数据集包含由GPT-3.5与GPT-4合成生成的短篇故事，所有故事均采用有限词汇量创作。
该数据集的相关描述见于以下论文：https://arxiv.org/abs/2305.07759。
论文中提及的模型均以TinyStories-train.txt作为训练集（tinystories-valid.txt文件可用于验证损失计算）。上述模型可在Huggingface平台的roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M仓库中获取。
附加资源如下：
1. tinystories_all_data.tar.gz：包含该类故事的超集数据集，附带元数据以及用于生成每篇故事的提示词。
2. TinyStoriesV2-GPT4-train.txt：该数据集的新版本，仅基于GPT-4生成的故事构建（原始数据集同时包含GPT-3.5生成的内容，质量相对较低）。其包含原始数据集中所有GPT-4生成的故事作为子集，但整体规模显著更大。
3. Evaluation_prompts.yaml：用于评估模型的提示词列表（详见论文）。