Bag of Lies: Robustness in Continuous Pre-training BERT

This is the dataset accompanying the paper "Bag of Lies: Robustness in Continuous Pre-training BERT". 

In this work, we evaluate to what extent entity knowledge about COVID-19 can be learned by BERT through continous pre-training, and how robust this process is. The two files in this dataset (small: litcov200 and large: litcov10K) are used to continue pre-training BERT. There are three variants in each file: "LitCovid original", which are texts extracted from the LitCovid repositor; "LitCovid falsified", which are texts in which the truthfulness of the original is reversed using GPT-4Turbo; and "LitCovid paraphrased", in which the original texts are paraphrased using GPT-4Turbo.

本数据集配套论文《Bag of Lies：连续预训练BERT的鲁棒性》。
本研究旨在评估BERT（Bidirectional Encoder Representations from Transformers）通过持续预训练可习得的COVID-19（新型冠状病毒肺炎）实体知识程度，以及该预训练过程的鲁棒性。
本数据集包含两个文件，分别为小体量版本litcov200与大体量版本litcov10K，二者均用于BERT的持续预训练。
每个文件均包含三种变体：
1. "LitCovid original（LitCovid原始文本）"：从LitCovid（新冠文献数据库）中提取的原始文本；
2. "LitCovid falsified（LitCovid伪造文本）"：通过GPT-4Turbo反转原始文本真实性生成的文本；
3. "LitCovid paraphrased（LitCovid释义文本）"：通过GPT-4Turbo对原始文本进行释义改写生成的文本。