RyanSaklad/ReCast
收藏资源简介：
--- license: cc-by-4.0 task_categories: - other language: - en tags: - causal-inference - causal-graphs - benchmark - nlp - text-to-graph pretty_name: ReCast - Real-world Causal Graph Generation Benchmark size_categories: - n<1K configs: - config_name: default data_files: - split: test path: test.parquet default: true - config_name: responses data_files: - split: test path: responses.parquet - config_name: evaluations data_files: - split: test path: evaluations.parquet --- # ReCast: Real-world Causal Graph Generation Benchmark ReCast is a benchmark for evaluating LLMs on causal graph extraction from real-world scientific text. It contains 292 expert-annotated causal graphs from peer-reviewed economics and public policy research articles. **Paper:** [Can Large Language Models Infer Causal Relationships from Real-World Text?](https://arxiv.org/abs/2505.18931) **GitHub:** [ReCast Repository](https://github.com/Ryan-Saklad/ReCast) ## Dataset Configurations This dataset has three configurations: ### `default` The core benchmark with 292 causal graph samples. ```python from datasets import load_dataset ds = load_dataset("RyanSaklad/ReCast", split="test") ``` ### `responses` Model responses from various LLMs on the benchmark tasks. ```python ds = load_dataset("RyanSaklad/ReCast", "responses", split="test") ``` ### `evaluations` Evaluation scores for model responses using multiple evaluation methods. ```python ds = load_dataset("RyanSaklad/ReCast", "evaluations", split="test") ``` ## Benchmark Schema | Field | Type | Description | |-------|------|-------------| | `id` | int | Unique sample identifier | | `title` | string | Article title | | `source` | string | Publisher (PLOS or MDPI) | | `url` | string | Article URL (DOI for PLOS, article page for MDPI) | | `domains` | list[str] | Research domains/categories | | `num_nodes` | int | Number of nodes in the causal graph | | `num_edges` | int | Number of edges in the causal graph | | `explicitness` | float | Proportion of nodes explicitly mentioned in text (0-1) | | `nodes` | list[str] | List of node names | | `edges` | list[object] | List of edges with `source` and `target` fields | | `node_explicitness` | string (JSON) | Per-node explicitness scores (1=explicit, 2=implicit, 3=absent) | | `input_text` | string | Source text for causal graph extraction | | `abstract` | string | Article abstract | | `publication_date` | string | Publication date | ## Responses Schema | Field | Type | Description | |-------|------|-------------| | `id` | int | Response identifier | | `sample_id` | int | Foreign key to benchmark sample | | `model` | string | Model name (e.g., "deepseek/deepseek-r1") | | `task_type` | string | Task type (causal_graph_generation or causal_graph_generation_with_node_names) | | `response_answer` | string | Model's generated answer | | `response_reasoning` | string | Model's chain-of-thought reasoning (if available) | | `corrected_answer` | string | Human-corrected answer (if applicable) | | `valid_format` | int | Whether response was valid JSON format | | `response_date` | string | When response was generated | ## Evaluations Schema | Field | Type | Description | |-------|------|-------------| | `id` | int | Evaluation identifier | | `response_id` | int | Foreign key to response | | `sample_id` | int | Foreign key to benchmark sample | | `model` | string | Model that generated the response | | `task_type` | string | Task type | | `evaluator_type` | string | Evaluation method (see Evaluator Types below) | | `score` | float | Evaluation score (interpretation depends on evaluator_type) | | `evaluation_answer` | string | Detailed evaluation output | | `evaluation_reasoning` | string | Evaluator reasoning | | `evaluation_date` | string | When evaluation was performed | ## Evaluator Types The `evaluator_type` field indicates which evaluation method was used: | Evaluator | Description | Score | evaluation_answer | |-----------|-------------|-------|-------------------| | `graph_similarity` | Deterministic graph matching using normalized Structural Hamming Distance | 0-1 (higher = more similar) | Empty | | `llm_judge_deepseek/deepseek-r1` | LLM-as-judge providing 3 aggregate scores | 0 (see JSON) | JSON with 1-5 scale scores | | `fine_grained` | Per-node/per-edge LLM evaluation with detailed labels | 0-1 composite | YAML with detailed labels | ### LLM Judge Format The `llm_judge_deepseek/deepseek-r1` evaluations contain aggregate scores in JSON: ```json { "scores": { "causal_accuracy": 3, "causal_recall": 2, "semantic_similarity": 3 } } ``` Each score is on a 1-5 scale where 5 is best. ### Fine-Grained Evaluation Format The `fine_grained` evaluations provide detailed per-node and per-edge assessments in YAML format. This enables computing precision, recall, and F1 scores at both node and edge levels. ```yaml node_precision_evaluations: - node_number: 1 graph_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_STRONG | SEMANTIC_MODERATE | SEMANTIC_WEAK | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA text_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_STRONG | SEMANTIC_MODERATE | SEMANTIC_WEAK | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA node_recall_evaluations: - node_number: 1 importance_label: IMPORTANCE_CORE | IMPORTANCE_INTERMEDIATE | IMPORTANCE_PERIPHERAL presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_COMPLETE | SEMANTIC_PARTIAL | SEMANTIC_MINIMAL | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA edge_precision_evaluations: - edge_number: 1 graph_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH directionality_label: DIRECTION_CORRECT | DIRECTION_REVERSED | DIRECTION_NA abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA text_evaluation: presence_label: PRESENCE_GRAPH_ONLY | PRESENCE_EXPLICIT | PRESENCE_IMPLIED | PRESENCE_NO_MATCH inference_label: INFERENCE_DIRECT | INFERENCE_DERIVED | INFERENCE_STRETCHED | INFERENCE_NA abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA edge_recall_evaluations: - edge_number: 1 importance_label: IMPORTANCE_CENTRAL | IMPORTANCE_CONNECTING | IMPORTANCE_AUXILIARY presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH directionality_label: DIRECTION_CORRECT | DIRECTION_REVERSED | DIRECTION_UNCLEAR | DIRECTION_MISSING abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA ``` #### Label Definitions **Presence Labels:** - `STRONG_MATCH`: Core concept matches with only minor differences - `WEAK_MATCH`: Core concept shares meaning but has notable differences - `NO_MATCH`: No corresponding concept found **Semantic Labels (Precision):** - `STRONG`: Exactly or nearly identical meaning - `MODERATE`: Same core concept with meaningful differences in scope - `WEAK`: Shares some semantic space but with substantial differences **Semantic Labels (Recall):** - `COMPLETE`: Ground-truth concept fully captured - `PARTIAL`: Core aspects captured with some meaning loss - `MINIMAL`: Only surface-level aspects captured **Abstraction Labels:** - `BROADER`: More general concept than ground truth - `ALIGNED`: Same scope and specificity - `NARROWER`: More specific subset of ground truth **Importance Labels:** - `CORE` / `CENTRAL`: Fundamental to the causal structure - `INTERMEDIATE` / `CONNECTING`: Key connection between concepts - `PERIPHERAL` / `AUXILIARY`: Supplementary information **Directionality Labels:** - `CORRECT`: Causality direction matches - `REVERSED`: Causality direction is opposite - `UNCLEAR`: Direction is ambiguous - `MISSING`: Relationship absent **Inference Labels:** - `DIRECT`: Matches explicit causal claims in text - `DERIVED`: Logically follows from text - `STRETCHED`: Possible but weakly supported ## Models Included Responses and evaluations are included from: - DeepSeek R1 (`deepseek/deepseek-r1`) - OpenAI o3-mini (`openai/o3-mini`) - Qwen 2.5 32B (`qwen/qwen2.5-32b-instruct`) - Qwen QwQ 32B (`qwen/qwq-32b`) - Llama 3.1 8B (`meta-llama/llama-3.1-8b-instruct`) ## Citation ```bibtex @misc{saklad2025largelanguagemodelsinfer, title={Can Large Language Models Infer Causal Relationships from Real-World Text?}, author={Ryan Saklad and Aman Chadha and Oleg Pavlov and Raha Moraffah}, year={2025}, eprint={2505.18931}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.18931}, } ``` ## License This dataset is released under CC-BY 4.0. The source articles are from open-access journals (PLOS and MDPI) published under CC-BY licenses.
--- license: 知识共享署名4.0（CC BY 4.0） task_categories: - 其他 language: - 英语 tags: - 因果推断（causal-inference） - 因果图（causal-graphs） - 基准测试集（benchmark） - 自然语言处理（NLP） - 文本到图（text-to-graph） pretty_name: ReCast——真实世界因果图生成基准测试集 size_categories: - n<1000 configs: - config_name: default data_files: - split: test path: test.parquet default: true - config_name: responses data_files: - split: test path: responses.parquet - config_name: evaluations data_files: - split: test path: evaluations.parquet --- # ReCast：真实世界因果图生成基准测试集 ReCast是一款用于评估大语言模型（Large Language Model, LLM）从真实世界科学文本中提取因果图的基准测试集。它包含292份经专家标注的因果图，均来自经过同行评审的经济学与公共政策研究论文。 **论文**：[《大语言模型能否从真实世界文本中推断因果关系？》](https://arxiv.org/abs/2505.18931) **GitHub仓库**：[ReCast 仓库](https://github.com/Ryan-Saklad/ReCast) ## 数据集配置 本数据集包含三种配置项： ### `default`（默认配置） 包含292个因果图样本的核心基准集。 python from datasets import load_dataset ds = load_dataset("RyanSaklad/ReCast", split="test") ### `responses`（响应配置） 收录各大语言模型在本基准任务上生成的模型响应。 python ds = load_dataset("RyanSaklad/ReCast", "responses", split="test") ### `evaluations`（评估配置） 收录使用多种评估方法对模型响应生成的评估分数。 python ds = load_dataset("RyanSaklad/ReCast", "evaluations", split="test") ## 基准集字段说明 | 字段名 | 数据类型 | 字段描述 | |-------|----------|----------| | `id` | 整数（int） | 唯一样本标识符 | | `title` | 字符串（string） | 文章标题 | | `source` | 字符串（string） | 出版方（PLOS 或 MDPI） | | `url` | 字符串（string） | 文章链接（PLOS 为 DOI，MDPI 为文章页面地址） | | `domains` | 字符串列表（list[str]） | 研究领域/分类 | | `num_nodes` | 整数（int） | 因果图的节点数量 | | `num_edges` | 整数（int） | 因果图的边数量 | | `explicitness` | 浮点数（float） | 文本中明确提及的节点占比（取值范围 0~1） | | `nodes` | 字符串列表（list[str]） | 节点名称列表 | | `edges` | 对象列表（list[object]） | 边列表，每条边包含 `source` 和 `target` 字段 | | `node_explicitness` | JSON 格式字符串（string） | 按节点划分的明确性评分（1=明确提及，2=隐含提及，3=未提及） | | `input_text` | 字符串（string） | 用于因果图提取的源文本 | | `abstract` | 字符串（string） | 文章摘要 | | `publication_date` | 字符串（string） | 文章发表日期 | ## 响应字段说明 | 字段名 | 数据类型 | 字段描述 | |-------|----------|----------| | `id` | 整数（int） | 响应标识符 | | `sample_id` | 整数（int） | 关联基准集样本的外键 | | `model` | 字符串（string） | 模型名称（例如 "deepseek/deepseek-r1"） | | `task_type` | 字符串（string） | 任务类型（`causal_graph_generation` 或 `causal_graph_generation_with_node_names`） | | `response_answer` | 字符串（string） | 模型生成的答案 | | `response_reasoning` | 字符串（string） | 模型的思维链推理过程（若有） | | `corrected_answer` | 字符串（string） | 人工修正后的答案（若适用） | | `valid_format` | 整数（int） | 标识响应是否为有效 JSON 格式 | | `response_date` | 字符串（string） | 响应生成时间 | ## 评估字段说明 | 字段名 | 数据类型 | 字段描述 | |-------|----------|----------| | `id` | 整数（int） | 评估标识符 | | `response_id` | 整数（int） | 关联响应的外键 | | `sample_id` | 整数（int） | 关联基准集样本的外键 | | `model` | 字符串（string） | 生成对应响应的模型名称 | | `task_type` | 字符串（string） | 任务类型 | | `evaluator_type` | 字符串（string） | 评估方法（详见下方「评估器类型」说明） | | `score` | 浮点数（float） | 评估分数（具体含义取决于 `evaluator_type`） | | `evaluation_answer` | 字符串（string） | 详细的评估输出 | | `evaluation_reasoning` | 字符串（string） | 评估推理过程 | | `evaluation_date` | 字符串（string） | 评估执行时间 | ## 评估器类型 `evaluator_type` 字段用于指示所使用的评估方法： | 评估器名称 | 方法描述 | 分数范围 | `evaluation_answer` 说明 | |-----------|----------|----------|--------------------------| | `graph_similarity` | 使用归一化结构汉明距离的确定性图匹配方法 | 0~1（分数越高，相似度越高） | 空字符串 | | `llm_judge_deepseek/deepseek-r1` | 以大语言模型作为评判者，提供3项聚合评分 | 详见下方 JSON 格式说明，每项采用 1~5 分制，5分为最优 | 包含 1~5 分制评分的 JSON 字符串 | | `fine_grained` | 针对节点/边的细粒度大语言模型评估，附带详细标签 | 0~1 的复合得分 | 包含详细评估标签的 YAML 格式字符串 | ### LLM 评判者格式 `llm_judge_deepseek/deepseek-r1` 类型的评估结果以 JSON 格式存储聚合评分，示例如下： json { "scores": { "causal_accuracy": 3, "causal_recall": 2, "semantic_similarity": 3 } } 每项评分采用 1~5 分制，5分为最优。 ### 细粒度评估格式 `fine_grained` 类型的评估以 YAML 格式提供节点和边的详细评估结果，支持计算节点与边层面的精确率、召回率和 F1 值。示例如下： yaml node_precision_evaluations: - node_number: 1 graph_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_STRONG | SEMANTIC_MODERATE | SEMANTIC_WEAK | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA text_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_STRONG | SEMANTIC_MODERATE | SEMANTIC_WEAK | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA node_recall_evaluations: - node_number: 1 importance_label: IMPORTANCE_CORE | IMPORTANCE_INTERMEDIATE | IMPORTANCE_PERIPHERAL presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH semantic_label: SEMANTIC_COMPLETE | SEMANTIC_PARTIAL | SEMANTIC_MINIMAL | SEMANTIC_NA abstraction_label: ABSTRACTION_BROADER | ABSTRACTION_ALIGNED | ABSTRACTION_NARROWER | ABSTRACTION_NA edge_precision_evaluations: - edge_number: 1 graph_evaluation: presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH directionality_label: DIRECTION_CORRECT | DIRECTION_REVERSED | DIRECTION_NA abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA text_evaluation: presence_label: PRESENCE_GRAPH_ONLY | PRESENCE_EXPLICIT | PRESENCE_IMPLIED | PRESENCE_NO_MATCH inference_label: INFERENCE_DIRECT | INFERENCE_DERIVED | INFERENCE_STRETCHED | INFERENCE_NA abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA edge_recall_evaluations: - edge_number: 1 importance_label: IMPORTANCE_CENTRAL | IMPORTANCE_CONNECTING | IMPORTANCE_AUXILIARY presence_label: PRESENCE_STRONG_MATCH | PRESENCE_WEAK_MATCH | PRESENCE_NO_MATCH directionality_label: DIRECTION_CORRECT | DIRECTION_REVERSED | DIRECTION_UNCLEAR | DIRECTION_MISSING abstraction_label: ABSTRACTION_ALIGNED | ABSTRACTION_BROADER | ABSTRACTION_NARROWER | ABSTRACTION_NA #### 标签定义 **存在性标签（Presence Labels）**： - `STRONG_MATCH`：核心概念匹配，仅存在细微差异 - `WEAK_MATCH`：核心概念语义一致，但存在显著差异 - `NO_MATCH`：未找到对应概念 **语义标签（精确率维度，Precision）**： - `STRONG`：语义完全或几乎完全一致 - `MODERATE`：核心概念一致，但范围存在实质性差异 - `WEAK`：共享部分语义空间，但差异显著 **语义标签（召回率维度，Recall）**： - `COMPLETE`：完全覆盖真实概念 - `PARTIAL`：覆盖核心方面，但存在部分语义损失 - `MINIMAL`：仅捕捉表层信息 **抽象程度标签（Abstraction Labels）**： - `BROADER`：比真实概念更泛化 - `ALIGNED`：范围和特异性与真实概念一致 - `NARROWER`：比真实概念更具体的子集 **重要性标签（Importance Labels）**： - `CORE` / `CENTRAL`：因果结构的核心要素 - `INTERMEDIATE` / `CONNECTING`：连接不同概念的关键环节 - `PERIPHERAL` / `AUXILIARY`：补充性信息 **方向性标签（Directionality Labels）**： - `CORRECT`：因果方向与真实图一致 - `REVERSED`：因果方向与真实图相反 - `UNCLEAR`：因果方向模糊不清 - `MISSING`：因果关系缺失 **推理类型标签（Inference Labels）**： - `DIRECT`：与文本中的显式因果主张匹配 - `DERIVED`：可从文本中逻辑推导得出 - `STRETCHED`：存在可能性，但支撑性较弱 ## 包含的模型 响应与评估数据来自以下模型： - DeepSeek R1（`deepseek/deepseek-r1`） - OpenAI o3-mini（`openai/o3-mini`） - Qwen 2.5 32B（`qwen/qwen2.5-32b-instruct`） - Qwen QwQ 32B（`qwen/qwq-32b`） - Llama 3.1 8B（`meta-llama/llama-3.1-8b-instruct`） ## 引用格式 bibtex @misc{saklad2025largelanguagemodelsinfer, title={Can Large Language Models Infer Causal Relationships from Real-World Text?}, author={Ryan Saklad and Aman Chadha and Oleg Pavlov and Raha Moraffah}, year={2025}, eprint={2505.18931}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2505.18931}, } ## 许可协议 本数据集采用知识共享署名4.0（CC BY 4.0）许可协议发布。源文章均来自开放获取期刊PLOS与MDPI，且采用CC-BY许可协议发布。
