Dataset for the paper: "Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims"

Overview This dataset of medical misinformation was collected and is published by Kempelen Institute of Intelligent Technologies (KInIT). It consists of approx. 317k news articles and blog posts on medical topics published between January 1, 1998 and February 1, 2022 from a total of 207 reliable and unreliable sources. The dataset contains full-texts of the articles, their original source URL and other extracted metadata. If a source has a credibility score available (e.g., from Media Bias/Fact Check), it is also included in the form of annotation. Besides the articles, the dataset contains around 3.5k fact-checks and extracted verified medical claims with their unified veracity ratings published by fact-checking organisations such as Snopes or FullFact. Lastly and most importantly, the dataset contains 573 manually and more than 51k automatically labelled mappings between previously verified claims and the articles; mappings consist of two values: claim presence (i.e., whether a claim is contained in the given article) and article stance (i.e., whether the given article supports or rejects the claim or provides both sides of the argument). The dataset is primarily intended to be used as a training and evaluation set for machine learning methods for claim presence detection and article stance classification, but it enables a range of other misinformation related tasks, such as misinformation characterisation or analyses of misinformation spreading. Its novelty and our main contributions lie in (1) focus on medical news article and blog posts as opposed to social media posts or political discussions; (2) providing multiple modalities (beside full-texts of the articles, there are also images and videos), thus enabling research of multimodal approaches; (3) mapping of the articles to the fact-checked claims (with manual as well as predicted labels); (4) providing source credibility labels for 95% of all articles and other potential sources of weak labels that can be mined from the articles' content and metadata. The dataset is associated with the research paper "Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims" accepted and presented at ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). The accompanying Github repository provides a small static sample of the dataset and the dataset's descriptive analysis in a form of Jupyter notebooks. In order to obtain an access to the full dataset (in the CSV format), please, request the access by following the instructions provided below. Note: Please, check also our MultiClaim Dataset that provides a more recent, a larger, and a highly multilingual dataset of fact-checked claims, social media posts and relations between them. References If you use this dataset in any publication, project, tool or in any other form, please, cite the following papers: @inproceedings{SrbaMonantPlatform, author = {Srba, Ivan and Moro, Robert and Simko, Jakub and Sevcech, Jakub and Chuda, Daniela and Navrat, Pavol and Bielikova, Maria}, booktitle = {Proceedings of Workshop on Reducing Online Misinformation Exposure (ROME 2019)}, pages = {1--7}, title = {Monant: Universal and Extensible Platform for Monitoring, Detection and Mitigation of Antisocial Behavior}, year = {2019} } @inproceedings{SrbaMonantMedicalDataset, author = {Srba, Ivan and Pecher, Branislav and Tomlein Matus and Moro, Robert and Stefancova, Elena and Simko, Jakub and Bielikova, Maria}, booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22)}, numpages = {11}, title = {Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims}, year = {2022}, doi = {10.1145/3477495.3531726}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3477495.3531726}, } Dataset creation process In order to create this dataset (and to continuously obtain new data), we used our research platform Monant. The Monant platform provides so called data providers to extract news articles/blogs from news/blog sites as well as fact-checking articles from fact-checking sites. General parsers (from RSS feeds, Wordpress sites, Google Fact Check Tool, etc.) as well as custom crawler and parsers were implemented (e.g., for fact checking site Snopes.com). All data is stored in the unified format in a central data storage. Ethical considerations The dataset was collected and is published for research purposes only. We collected only publicly available content of news/blog articles. The dataset contains identities of authors of the articles if they were stated in the original source; we left this information, since the presence of an author's name can be a strong credibility indicator. However, we anonymised the identities of the authors of discussion posts included in the dataset. The main identified ethical issue related to the presented dataset lies in the risk of mislabelling of an article as supporting a false fact-checked claim and, to a lesser extent, in mislabelling an article as not containing a false claim or not supporting it when it actually does. To minimise these risks, we developed a labelling methodology and require an agreement of at least two independent annotators to assign a claim presence or article stance label to an article. It is also worth noting that we do not label an article as a whole as false or true. Nevertheless, we provide partial article-claim pair veracities based on the combination of claim presence and article stance labels. As to the veracity labels of the fact-checked claims and the credibility (reliability) labels of the articles' sources, we take these from the fact-checking sites and external listings such as Media Bias/Fact Check as they are and refer to their methodologies for more details on how they were established. Lastly, the dataset also contains automatically predicted labels of claim presence and article stance using our baselines described in the next section. These methods have their limitations and work with certain accuracy as reported in this paper. This should be taken into account when interpreting them. Reporting mistakes in the datasetThe mean to report considerable mistakes in raw collected data or in manual annotations is by creating a new issue in the accompanying Github repository. Alternately, general enquiries or requests can be sent at info [at] kinit.sk. Dataset structure Raw data At first, the dataset contains so called raw data (i.e., data extracted by the Web monitoring module of Monant platform and stored in exactly the same form as they appear at the original websites). Raw data consist of articles from news sites and blogs (e.g. naturalnews.com), discussions attached to such articles, fact-checking articles from fact-checking portals (e.g. snopes.com). In addition, the dataset contains feedback (number of likes, shares, comments) provided by user on social network Facebook which is regularly extracted for all news/blogs articles. Raw data are contained in these CSV files: sources.csv articles.csv article_media.csv article_authors.csv discussion_posts.csv discussion_post_authors.csv fact_checking_articles.csv fact_checking_article_media.csv claims.csv feedback_facebook.csv Note: Personal information about discussion posts' authors (name, website, gravatar) are anonymised. Annotations Secondly, the dataset contains so called annotations. Entity annotations describe the individual raw data entities (e.g., article, source). Relation annotations describe relation between two of such entities. Each annotation is described by the following attributes: category of annotation (`annotation_category`). Possible values: label (annotation corresponds to ground truth, determined by human experts) and prediction (annotation was created by means of AI method). type of annotation (`annotation_type_id`). Example values: Source reliability (binary), Claim presence. The list of possible values can be obtained from enumeration in annotation_types.csv. method which created annotation (`method_id`). Example values: Expert-based source reliability evaluation, Fact-checking article to claim transformation method. The list of possible values can be obtained from enumeration methods.csv. its value (`value`). The value is stored in JSON format and its structure differs according to particular annotation type. At the same time, annotations are associated with a particular object identified by: entity type (parameter `entity_type` in case of entity annotations, or `source_entity_type` and `target_entity_type` in case of relation annotations). Possible values: sources, articles, fact-checking-articles. entity id (parameter `entity_id` in case of entity annotations, or `source_entity_id` and `target_entity_id` in case of relation annotations). The dataset provides specifically these entity annotations: Source reliability (binary). Determines validity of source (website) at a binary scale with two options: reliable source and unreliable source. Article veracity. Aggregated information about veracity from article-claim pairs. The dataset provides specifically these relation annotations: Fact-checking article to claim mapping. Determines mapping between fact-checking article and claim. Claim presence. Determines presence of claim in article. Claim stance. Determines stance of an article to a claim. Annotations are contained in these CSV files: entity_annotations.csv relation_annotations.csv Note: Identification of human annotators authors (email provided in the annotation app) is anonymised. Enumerations Finally, the dataset provides additional CSV files with enumerations: media_types.csv source_types.csv annotation_types.csv methods.csvOverview 本医学错误信息数据集由肯佩伦智能技术研究所（Kempelen Institute of Intelligent Technologies, KInIT）收集并发布。数据集包含1998年1月1日至2022年2月1日期间发布的约31.7万篇医学主题新闻文章与博客文章，共计207个可靠与不可靠的信息来源。数据集包含文章全文、原始来源URL及其他提取得到的元数据。若信息来源具备可用的可信度评分（credibility score）（例如来自Media Bias/Fact Check），则会以标注形式一并收录。除文章外，数据集还包含约3500份事实核查（fact-check）报告，以及由Snopes、FullFact等事实核查机构发布的已验证医学主张及其统一真实性评级。最后也是最重要的是，数据集包含573份手动标注与超过5.1万份自动标注的映射关系，用于关联已验证主张与文章；此类映射包含两个属性：主张存在性（claim presence）（即给定文章中是否包含该主张）与文章立场（article stance）（即文章对该主张的态度为支持、反对，或同时呈现双方论点）。 本数据集主要用于训练与评估用于主张存在性检测（claim presence detection）及文章立场分类（article stance classification）的机器学习方法，同时也可支撑一系列其他与错误信息相关的任务，例如错误信息特征刻画（misinformation characterisation）或错误信息传播分析。 本数据集的创新性与核心贡献在于：（1）聚焦医学新闻文章与博客文章，而非社交媒体帖子或政治讨论；（2）提供多模态（multimodal）数据（除文章全文外，还包含图片与视频），可支撑多模态方法的相关研究；（3）实现文章与经事实核查主张的映射（包含人工标注与预测标签）；（4）为95%的文章提供来源可信度标签，以及可从文章内容与元数据中挖掘的其他弱标签（weak labels）潜在来源。 本数据集关联的研究论文为《Monant医学错误信息数据集：将文章映射至经事实核查的主张》，该论文已被ACM信息检索研究与发展国际大会（ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22）接收并展示。 配套的GitHub仓库提供了数据集的小型静态样本，以及以Jupyter笔记本（Jupyter notebooks）形式呈现的数据集描述性分析。 如需获取完整CSV格式数据集，请按照下方说明提交访问申请。 注意：请同时关注我们的MultiClaim数据集，该数据集包含更新、更大规模且支持多语言的经事实核查主张、社交媒体帖子及其关联关系。 ## 参考文献 若将本数据集用于任何出版物、项目、工具或其他场景，请引用以下论文： bibtex @inproceedings{SrbaMonantPlatform, author = {Srba, Ivan and Moro, Robert and Simko, Jakub and Sevcech, Jakub and Chuda, Daniela and Navrat, Pavol and Bielikova, Maria}, booktitle = {《减少在线错误信息暴露研讨会（ROME 2019）论文集》}, pages = {1--7}, title = {Monant：用于监测、检测与缓解反社会行为的通用可扩展平台}, year = {2019} } @inproceedings{SrbaMonantMedicalDataset, author = {Srba, Ivan and Pecher, Branislav and Tomlein Matus and Moro, Robert and Stefancova, Elena and Simko, Jakub and Bielikova, Maria}, booktitle = {第45届国际ACM信息检索研究与发展大会（SIGIR '22）论文集}, numpages = {11}, title = {Monant医学错误信息数据集：将文章映射至经事实核查的主张}, year = {2022}, doi = {10.1145/3477495.3531726}, publisher = {计算机学会（ACM）}, address = {美国纽约州纽约市}, url = {https://doi.org/10.1145/3477495.3531726}, } ## 数据集构建流程 为构建本数据集（并持续获取新数据），我们使用了自研的Monant研究平台。Monant平台提供了所谓的“数据提供者”模块，可从新闻/博客网站提取文章，以及从事实核查网站提取事实核查文章。我们实现了通用解析器（parser）（支持RSS订阅源（RSS feeds）、WordPress站点（WordPress sites）、谷歌事实核查工具（Google Fact Check Tool）等）与自定义爬虫（crawler）及解析器（parser）（例如针对事实核查网站Snopes.com）。所有数据均以统一格式存储于中央数据存储中。 ## 伦理考量 本数据集的收集与发布仅用于研究目的。我们仅收集了新闻/博客文章的公开可获取内容。若原始来源中包含文章作者的身份信息，我们会予以保留，因为作者姓名可作为强有力的可信度指标。但我们对数据集中讨论帖作者的身份进行了匿名化（anonymised）处理。 本数据集相关的主要伦理风险在于，存在将文章误标注为支持虚假经事实核查主张的可能，次要风险则是将实际包含虚假主张或支持该主张的文章误标注为不包含该主张或不支持该主张。为最小化此类风险，我们制定了标注方法，并要求至少两名独立标注者达成一致，才能为文章分配主张存在性（claim presence）或文章立场（article stance）标签。同时需要说明的是，我们不会对整篇文章直接标注为虚假或真实。尽管如此，我们仍基于主张存在性与文章立场标签的组合，提供了文章-主张对的部分真实性信息。 至于经事实核查主张的真实性标签与文章来源的可信度（可靠性）标签，我们直接采用了事实核查网站及Media Bias/Fact Check等外部列表的现有标注，具体标注方法的细节请参考其官方文档。 最后，数据集还包含使用我们在后续章节中描述的基线模型生成的主张存在性（claim presence）与文章立场（article stance）自动预测标签。此类方法存在自身局限性，且如论文中所述，仅能达到特定的准确率水平。在解读此类自动标注结果时，请务必考虑这一点。 ## 数据集错误上报 若您发现原始采集数据或手动标注中存在显著错误，请通过配套GitHub仓库提交新Issue进行反馈。此外，一般性咨询或请求可发送至info [at] kinit.sk。 ## 数据集结构 ### 原始数据 数据集首先包含所谓的“原始数据”，即由Monant平台的Web监控模块提取、且与原始网站上呈现的内容完全一致的数据。原始数据包含来自新闻网站与博客（如naturalnews.com）的文章、附属于此类文章的讨论，以及来自事实核查门户（如snopes.com）的事实核查文章。此外，数据集还包含所有新闻/博客文章定期从社交媒体平台Facebook提取的用户反馈数据（点赞数、分享数、评论数）。 原始数据存储于以下CSV文件中： - sources.csv - articles.csv - article_media.csv - article_authors.csv - discussion_posts.csv - discussion_post_authors.csv - fact_checking_articles.csv - fact_checking_article_media.csv - claims.csv - feedback_facebook.csv 注意：讨论帖作者的个人信息（姓名、网站、头像）均已匿名化。 ### 标注数据 其次，数据集包含所谓的“标注数据”。实体标注（entity annotations）用于描述单个原始数据实体（如文章、信息来源），关系标注（relation annotations）则用于描述此类实体间的关联关系。 每个标注均包含以下属性： 1. 标注类别（`annotation_category`）：可选值为`label`（标注为人工专家确定的真实标签（ground truth））与`prediction`（由AI方法生成的标注）。 2. 标注类型（`annotation_type_id`）：示例值包括“来源可信度（二元）”“主张存在性（claim presence）”，完整可选值列表可从`annotation_types.csv`的枚举项中获取。 3. 标注生成方法（`method_id`）：示例值包括“基于专家的来源可信度评估”“事实核查文章至主张的转换方法”，完整可选值列表可从`methods.csv`的枚举项中获取。 4. 标注值（`value`）：以JSON格式存储，其结构随标注类型的不同而有所差异。 同时，所有标注均关联至特定对象，该对象通过以下参数标识： - 实体类型：实体标注的参数为`entity_type`，关系标注的参数为`source_entity_type`与`target_entity_type`。可选值包括`sources`、`articles`、`fact-checking-articles`。 - 实体ID：实体标注的参数为`entity_id`，关系标注的参数为`source_entity_id`与`target_entity_id`。 数据集提供的特定实体标注包括： - 来源可信度（二元）：以二元尺度判定信息来源（网站）的有效性，可选值为“可靠来源”与“不可靠来源”。 - 文章真实性：基于文章-主张对聚合得到的真实性信息。 数据集提供的特定关系标注包括： - 事实核查文章与主张的映射：判定事实核查文章与主张之间的关联关系。 - 主张存在性（claim presence）：判定主张是否存在于文章中。 - 主张立场（article stance）：判定文章对某一主张的立场。 标注数据存储于以下CSV文件中： - entity_annotations.csv - relation_annotations.csv 注意：人工标注者的身份信息（标注应用中提供的邮箱）均已匿名化。 ## 枚举文件 最后，数据集提供了额外的枚举类CSV文件： - media_types.csv - source_types.csv - annotation_types.csv - methods.csv