EleutherAI/deep-ignorance-pretraining-mix

Deep Ignorance模型套件是一个由6.9B模型组成的集合，旨在促进对预训练、可解释性、训练数据和遗忘的研究。它包含了18个模型，其中一个是基于未过滤数据的基线模型，其余17个模型是基于过滤数据集或应用了其他安全干预措施训练的。预训练阶段的模型有101个检查点，退火阶段有11个。

The Deep Ignorance Model Suite is a collection of 6.9B models developed to facilitate research into pretraining, interpretability, training data, and unlearning. It contains 18 models composing of a baseline model trained on unfiltered data, and 17 models trained on filtered datasets or with other safety interventions being applied. Pretraining stage models have 101 checkpoints and annealing stage have 11.