IKMLab-team/cfever

--- license: apache-2.0 configs: - config_name: default data_files: - split: train path: train*.jsonl - split: dev path: dev*.jsonl - split: test path: test*.jsonl - config_name: train data_files: - split: train path: train*.jsonl - config_name: dev data_files: - split: dev path: dev*.jsonl - config_name: test data_files: - split: test path: test*.jsonl - config_name: wiki_pages data_files: - split: plain path: wiki*.jsonl --- # CFEVER-data ## Introduction to CFEVER This repository contains the dataset for our AAAI 2024 paper, "CFEVER: A Chinese Fact Extraction and VERification Dataset". [Paper link](https://doi.org/10.1609/aaai.v38i17.29825). ## Leaderboard website Please visit https://ikmlab.github.io/CFEVER to check the leaderboard of CFEVER. ## How to load CFEVER ```python from datasets import load_dataset # Get ready data ["train", "dev", "test"] ds = load_dataset("IKMLab-team/cfever") # Get separted data by: train, dev, test = ds["train"], ds["dev"], ds["test"] # If you only want a part of data train = load_dataset("IKMLab-team/cfever", name="train")["train"] # or train = load_dataset("IKMLab-team/cfever", split="train") # wiki page data: wiki_pages = load_dataset("IKMLab-team/cfever", name="wiki_pages")["plain"] ``` ## Repository structure ``` CFEVER-data ├── dev.jsonl # CFEVER development set ├── test.jsonl # CFEVER test set without labels and evidence ├── train.jsonl # CFEVER training set ├── wiki*.jsonl # CFEVER wiki pages ├── LICENSE ├── README.md └── sample_submission.jsonl # sample submission file of the test set ``` ## Evaluation - Please refer to our codebase: https://github.com/IKMLab/CFEVER-baselines/?tab=readme-ov-file#evaluations ## Submission - Please include three fields (necessary) in the prediction file for each claim in the test set. - `id` - `predicted_label` - `predicted_evidence` - The `id` field has been already included in [the test set](data/test.jsonl). Please do not change the order. - The `predicted_label` should be one of `supports`, `refutes`, or `NOT ENOUGH INFO`. - The `predicted_evidence` should be a list of evidence sentences, where each evidence sentence is represented by a list of `[page_id, line_number]`. For example: ``` # One evidence sentence for the claim { "id": 1, "predicted_label": "REFUTES", "predicted_evidence": [ ["page_id_2", 2], ] } ``` ``` # Two evidence sentences for the claim { "id": 1, "predicted_label": "SUPPORTS", "predicted_evidence": [ ["page_id_1", 1], ["page_id_2", 2], ] } ``` ``` # The claim cannot be verified { "id": 1, "predicted_label": "NOT ENOUGH INFO", "predicted_evidence": None } ``` - After creating the prediction file, please email the file to [e-mail] with a brief description of your method. We will evaluate your submission and update the leaderboard. - A randomly generated submission file can be found [here](sample_submission.jsonl). - Note that `claim` is not necessary to be included in the submission file. - You can also check [the prediction example for the development set](https://github.com/IKMLab/CFEVER-baselines/blob/main/simple_baseline/data/dumb_dev_pred.jsonl) and follow [the evaluation steps](https://github.com/IKMLab/CFEVER-baselines/tree/main?tab=readme-ov-file#sentence-retrieval-and-claim-verification) from [our CFEVER-baselines repo](https://github.com/IKMLab/CFEVER-baselines). ## Licensing Information CFEVER's data annotations incorporate content from Wikipedia, which is licensed under the Wikipedia Copyright Policy. Users of this dataset are responsible for ensuring that their use, redistribution, and downstream applications comply with all applicable licenses and attribution requirements of the Wikipedia license terms. ## Reference If you find our work useful, please cite our paper. ``` @article{Lin_Lin_Yeh_Li_Hu_Hsu_Lee_Kao_2024, title = {CFEVER: A Chinese Fact Extraction and VERification Dataset}, author = {Lin, Ying-Jia and Lin, Chun-Yi and Yeh, Chia-Jen and Li, Yi-Ting and Hu, Yun-Yu and Hsu, Chih-Hao and Lee, Mei-Feng and Kao, Hung-Yu}, doi = {10.1609/aaai.v38i17.29825}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, month = {Mar.}, number = {17}, pages = {18626-18634}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/29825}, volume = {38}, year = {2024}, bdsk-url-1 = {https://ojs.aaai.org/index.php/AAAI/article/view/29825}, bdsk-url-2 = {https://doi.org/10.1609/aaai.v38i17.29825} } ```许可证：Apache-2.0 配置项： - 配置名称：default 数据文件： - 数据集划分：train 路径：train*.jsonl - 数据集划分：dev 路径：dev*.jsonl - 数据集划分：test 路径：test*.jsonl - 配置名称：train 数据文件： - 数据集划分：train 路径：train*.jsonl - 配置名称：dev 数据文件： - 数据集划分：dev 路径：dev*.jsonl - 配置名称：test 数据文件： - 数据集划分：test 路径：test*.jsonl - 配置名称：wiki_pages 数据文件： - 数据集划分：plain 路径：wiki*.jsonl # CFEVER数据集 ## CFEVER数据集简介 本仓库包含我们发表于AAAI 2024的论文《CFEVER：中文事实抽取与验证数据集》所使用的数据集。[论文链接](https://doi.org/10.1609/aaai.v38i17.29825)。 ## 排行榜站点 请访问 https://ikmlab.github.io/CFEVER 以查看CFEVER的排行榜。 ## CFEVER数据集加载方法 python from datasets import load_dataset # 获取训练、验证、测试全量数据集 ds = load_dataset("IKMLab-team/cfever") # 按划分获取独立数据集： train, dev, test = ds["train"], ds["dev"], ds["test"] # 若仅需部分数据 train = load_dataset("IKMLab-team/cfever", name="train")["train"] # 或 train = load_dataset("IKMLab-team/cfever", split="train") # 维基页面数据： wiki_pages = load_dataset("IKMLab-team/cfever", name="wiki_pages")["plain"] ## 仓库结构 CFEVER-data ├── dev.jsonl # CFEVER验证集 ├── test.jsonl # 无标签与证据的CFEVER测试集 ├── train.jsonl # CFEVER训练集 ├── wiki*.jsonl # CFEVER维基页面数据 ├── LICENSE # 许可证文件 ├── README.md # 说明文档 └── sample_submission.jsonl # 测试集示例提交文件 ## 评估 请参考我们的代码库：https://github.com/IKMLab/CFEVER-baselines/?tab=readme-ov-file#evaluations ## 提交要求 - 针对测试集的每条声明，预测文件必须包含以下三个必填字段： - `id` - `predicted_label`（预测标签） - `predicted_evidence`（预测证据） - `id`字段已包含在[测试集](data/test.jsonl)中，请不要更改数据顺序。 - `predicted_label`的可选取值为`supports`（支持）、`refutes`（反驳）或`NOT ENOUGH INFO`（信息不足）。 - `predicted_evidence`应为证据句子列表，每条证据句子以`[page_id, line_number]`形式的列表表示。例如： # 单条证据的声明示例 { "id": 1, "predicted_label": "REFUTES", "predicted_evidence": [ ["page_id_2", 2], ] } # 两条证据的声明示例 { "id": 1, "predicted_label": "SUPPORTS", "predicted_evidence": [ ["page_id_1", 1], ["page_id_2", 2], ] } # 无法验证的声明示例 { "id": 1, "predicted_label": "NOT ENOUGH INFO", "predicted_evidence": None } - 生成预测文件后，请将文件发送至[e-mail]，并附带您所使用方法的简要说明。我们将对您的提交进行评估并更新排行榜。 - 随机生成的示例提交文件可在[此处](sample_submission.jsonl)获取。 - 注意：提交文件中无需包含`claim`（声明）字段。 您也可以参考[验证集预测示例](https://github.com/IKMLab/CFEVER-baselines/blob/main/simple_baseline/data/dumb_dev_pred.jsonl)，并遵循[我们的CFEVER基线仓库](https://github.com/IKMLab/CFEVER-baselines)中的[评估步骤](https://github.com/IKMLab/CFEVER-baselines/tree/main?tab=readme-ov-file#sentence-retrieval-and-claim-verification)。 ## 许可信息 CFEVER数据集的标注内容包含来自维基百科的内容，该内容遵循维基百科版权政策。本数据集的使用者需确保其使用、再分发及下游应用均符合维基百科许可条款的所有适用许可要求与归因规范。 ## 引用 如果您认为我们的工作对您有所帮助，请引用我们的论文。 @article{Lin_Lin_Yeh_Li_Hu_Hsu_Lee_Kao_2024, title = {CFEVER: A Chinese Fact Extraction and VERification Dataset}, author = {Lin, Ying-Jia and Lin, Chun-Yi and Yeh, Chia-Jen and Li, Yi-Ting and Hu, Yun-Yu and Hsu, Chih-Hao and Lee, Mei-Feng and Kao, Hung-Yu}, doi = {10.1609/aaai.v38i17.29825}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, month = {Mar.}, number = {17}, pages = {18626-18634}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/29825}, volume = {38}, year = {2024}, bdsk-url-1 = {https://ojs.aaai.org/index.php/AAAI/article/view/29825}, bdsk-url-2 = {https://doi.org/10.1609/aaai.v38i17.29825} }