TNSA/PT-LMD

RedPajama是一个干净、完全开源的LLaMa数据集实现。该数据集包含来自多个来源的1.2万亿令牌，包括Commoncrawl（8780亿）、C4（1750亿）、GitHub（590亿）、ArXiv（280亿）、Wikipedia（240亿）和StackExchange（200亿）。数据集主要由英语组成，尽管Wikipedia部分包含多种语言。数据集结构包括文本和元数据，如URL、时间戳、来源和语言等。创建该数据集的目的是尽可能遵循LLaMa论文的配方，以重现其方法。RedPajama is a clean-room, fully open-source implementation of the LLaMa dataset. The dataset consists of 1.2 trillion tokens from various sources including Commoncrawl (878 billion), C4 (175 billion), GitHub (59 billion), ArXiv (28 billion), Wikipedia (24 billion), and StackExchange (20 billion). The dataset is primarily in English, though the Wikipedia slice contains multiple languages. The dataset structure includes text and metadata such as URL, timestamp, source, and language. The dataset was created to follow the LLaMa paper as closely as possible to try to reproduce its recipe.