Data for: Evol-Preference : Evol-Preference: Automatic Evolution of Preference Data For Safety Alignment

The PKU-Alignment team released the dataset "Beavertails", which focuses on AI safety. We have extended and optimized "Beavertails" to obtain this dataset. Readers can directly use our dataset to train large language models to enhance their usefulness and harmlessness.Training details: 70% for supervised fine-tuning(SFT), 30% direct preference optimization (DPO), training hyperparameters available in Appendix C of the paper.