Bag of Lies: Robustness in Continuous Pre-training BERT

This is the dataset accompanying the paper "Bag of Lies: Robustness in Continuous Pre-training BERT". In this work, we evaluate to what extent entity knowledge about COVID-19 can be learned by BERT through continous pre-training, and how robust this process is. The two files in this dataset (small: litcov200 and large: litcov10K) are used to continue pre-training BERT. There are three variants in each file: "LitCovid original", which are texts extracted from the LitCovid repositor; "LitCovid falsified", which are texts in which the truthfulness of the original is reversed using GPT-4Turbo; and "LitCovid paraphrased", in which the original texts are paraphrased using GPT-4Turbo.