Proxy Models Dataset

该数据集是通过在包含1亿个标记的不同数据混合上训练小型代理模型（每个模型拥有100万个参数）而生成的。它反映了广泛的数据混合范围，包括每个领域从0%到100%的极端权重，确保了基于标记分布的统计代表性。这些小型代理模型（拥有100万个参数）在1亿个标记上进行训练，其任务是回归分析，旨在优化数据混合，以提高语言模型的性能。

This dataset is generated by training small proxy models (each with 1 million parameters) on diverse data mixtures containing 100 million tokens. It covers a wide spectrum of data mixtures, with extreme weights ranging from 0% to 100% for each domain, ensuring statistical representativeness grounded in token distributions. These small proxy models (with 1 million parameters) are trained on 100 million tokens using a regression task, with the goal of optimizing data mixtures to enhance the performance of language models.