llm-agents/CriticBench
收藏资源简介：
--- license: mit task_categories: - question-answering - text-classification - text-generation language: - en paperswithcode_id: criticbench pretty_name: CriticBench size_categories: - 1K<n<10K tags: - llm - reasoning - critique - correction - discrimination - math-word-problems - math-reasoning - question-answering - commonsense-reasoning - code-generation - symbolic-reasoning - algorithmic-reasoning --- # Dataset Card for Dataset Name <!-- Provide a quick summary of the dataset. --> CriticBench is a comprehensive benchmark designed to assess LLMs' abilities to generate, critique/discriminate and correct reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. ## Dataset Details ### Dataset Description <!-- Provide a longer summary of what this dataset is. --> - **Curated by:** THU - **Funded by [optional]:** [More Information Needed] - **Shared by [optional]:** [More Information Needed] - **Language(s) (NLP):** EN - **License:** MIT ### Dataset Sources [optional] <!-- Provide the basic links for the dataset. --> - **Repository:** https://github.com/CriticBench/CriticBench - **Paper [optional]:** https://arxiv.org/pdf/2402.14809.pdf - **Demo [optional]:** https://criticbench.github.io/ ## Uses <!-- Address questions around how the dataset is intended to be used. --> ### Direct Use <!-- This section describes suitable use cases for the dataset. --> [More Information Needed] ## Dataset Structure <!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. --> [More Information Needed] ## Dataset Creation ### Curation Rationale <!-- Motivation for the creation of this dataset. --> [More Information Needed] ### Source Data <!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). --> #### Data Collection and Processing <!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. --> [More Information Needed] #### Who are the source data producers? <!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. --> [More Information Needed] ### Annotations [optional] <!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. --> #### Annotation process <!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. --> [More Information Needed] #### Who are the annotators? <!-- This section describes the people or systems who created the annotations. --> [More Information Needed] #### Personal and Sensitive Information <!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. --> [More Information Needed] ## Bias, Risks, and Limitations <!-- This section is meant to convey both technical and sociotechnical limitations. --> [More Information Needed] ### Recommendations <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations. ## Citation [optional] <!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. --> **BibTeX:** ``` @misc{lin2024criticbench, title={CriticBench: Benchmarking LLMs for Critique-Correct Reasoning}, author={Zicheng Lin and Zhibin Gou and Tian Liang and Ruilin Luo and Haowei Liu and Yujiu Yang}, year={2024}, eprint={2402.14809}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` **APA:** Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., & Yang, Y. (2024). CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. ## Glossary [optional] <!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. --> [More Information Needed] ## More Information [optional] [More Information Needed] ## Dataset Card Authors [optional] [More Information Needed] ## Dataset Card Contact [More Information Needed]
许可证: MIT许可证 任务类别: - 问答 - 文本分类 - 文本生成 语言: - 英语 PapersWithCode编号: criticbench 友好名称: CriticBench 样本规模类别: - 1000 < 样本量 < 10000 标签: - 大语言模型（LLM） - 推理 - 评判 - 修正 - 判别 - 数学应用题 - 数学推理 - 问答 - 常识推理 - 代码生成 - 符号推理 - 算法推理 # 数据集卡片 CriticBench是一款综合性基准测试集，旨在评估大语言模型（LLM）在多种任务场景下的生成、评判/判别与推理修正能力。CriticBench涵盖五大推理领域：数学推理、常识推理、符号推理、代码生成与算法推理。该数据集整合了15个子数据集，并纳入了三类大语言模型家族的生成结果。 ## 数据集详情 ### 数据集概述 - **整理方**：清华大学（THU） - **资助方 [可选]**：[需补充更多信息] - **共享方 [可选]**：[需补充更多信息] - **自然语言处理所用语言**：英语 - **许可证**：MIT许可证 ### 数据集来源 [可选] - **代码仓库**：https://github.com/CriticBench/CriticBench - **相关论文 [可选]**：https://arxiv.org/pdf/2402.14809.pdf - **演示页面 [可选]**：https://criticbench.github.io/ ## 使用场景 ### 直接使用 [需补充更多信息] ## 数据集结构 [需补充更多信息] ## 数据集构建 ### 构建初衷 [需补充更多信息] ### 源数据 #### 数据收集与处理流程 [需补充更多信息] #### 源数据生产者 [需补充更多信息] ### 标注信息 [可选] #### 标注流程 [需补充更多信息] #### 标注者 [需补充更多信息] #### 个人与敏感信息说明 [需补充更多信息] ## 偏差、风险与局限性 [需补充更多信息] ### 建议 用户应知晓该数据集存在的风险、偏差与局限性，相关建议仍需进一步补充完善。 ## 引用信息 [可选] **BibTeX格式引用**： @misc{lin2024criticbench, title={CriticBench: Benchmarking LLMs for Critique-Correct Reasoning}, author={Zicheng Lin and Zhibin Gou and Tian Liang and Ruilin Luo and Haowei Liu and Yujiu Yang}, year={2024}, eprint={2402.14809}, archivePrefix={arXiv}, primaryClass={cs.CL} } **APA格式引用**： Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., & Yang, Y. (2024). CriticBench: Benchmarking LLMs for Critique-Correct Reasoning. ## 术语表 [可选] [需补充更多信息] ## 更多信息 [可选] [需补充更多信息] ## 数据集卡片撰写者 [可选] [需补充更多信息] ## 数据集卡片联系人 [需补充更多信息]
数据集卡片 for CriticBench
数据集概述
CriticBench 是一个综合基准，旨在评估大型语言模型（LLMs）在生成、批判/区分和纠正推理方面的能力，涵盖五个推理领域：数学、常识、符号、编码和算法。它整合了15个数据集，并包含了来自三个LLM家族的响应。
数据集详情
数据集描述
- 由：THU 策划
- 语言：英语（EN）
- 许可证：MIT
数据集来源
- 仓库：https://github.com/CriticBench/CriticBench
- 论文：https://arxiv.org/pdf/2402.14809.pdf
- 演示：https://criticbench.github.io/
使用
直接使用
[更多信息待补充]
数据集结构
[更多信息待补充]
数据集创建
策划理由
[更多信息待补充]
源数据
数据收集和处理
[更多信息待补充]
源数据生产者
[更多信息待补充]
标注
标注过程
[更多信息待补充]
标注者
[更多信息待补充]
个人和敏感信息
[更多信息待补充]
偏差、风险和局限性
[更多信息待补充]
建议
用户应意识到数据集的风险、偏差和局限性。更多信息待补充以供进一步建议。
引用
BibTeX:
@misc{lin2024criticbench, title={CriticBench: Benchmarking LLMs for Critique-Correct Reasoning}, author={Zicheng Lin and Zhibin Gou and Tian Liang and Ruilin Luo and Haowei Liu and Yujiu Yang}, year={2024}, eprint={2402.14809}, archivePrefix={arXiv}, primaryClass={cs.CL} }
APA:
Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., & Yang, Y. (2024). CriticBench: Benchmarking LLMs for Critique-Correct Reasoning.
