jhu-clsp/ettin-decay-data

Ettin衰减阶段数据集是用于训练Ettin编码器和解码器模型的数据集，包含高质量的长期内容和教育材料。数据集由多种数据源组成，包括最高质量的网络爬取数据、优质代码库、书籍、数学内容、高质量问答内容、学术预印本、百科全书文章和教科书。数据集在训练特征上包括积极的学习率衰减、长上下文、较低的掩码比例和注重质量而非数量的数据源。

The Ettin Decay Phase Data is a dataset used for training Ettin encoder and decoder models, containing high-quality long-form content and educational materials. The dataset is composed of various sources, including the highest-quality web crawl data, premium code repositories, books, mathematical content, high-quality Q&A content, academic preprints, encyclopedia articles, and textbooks. The dataset features aggressive learning rate decay, long context, lower masking ratio, and a focus on quality over quantity in its training characteristics.