jhu-clsp/ettin-extension-data

Ettin中训数据集是用于Ettin编码器和解码器模型中训阶段的数据集，由高质量过滤的Web爬虫、代码库、数学内容、科技论文、社交讨论、学术预印本、问答论坛、指导性数据、文学参考书籍和百科全书文章等组成。数据集在质量和技术上进行了多项改进，包括使用Dolmino过滤、增加上下文长度、更新位置编码和学习率衰减计划。

The Ettin Mid-training Data is a dataset used for the mid-training phase of Ettin encoder and decoder models, consisting of high-quality filtered web crawl, code repositories, mathematical content, scientific papers, social discussion threads, academic preprints, Q&A forums, instruction-following examples, literature and reference books, and encyclopedia articles. The dataset has undergone several quality and technical improvements, including Dolmino filtering, context length extension, updated positional encoding, and learning rate decay schedule.