ProverQA

ProverQA数据集，主要用于评测大语言模型LLMs的逻辑推理能力。有训练集和评测集，训练集5000个example，评测集1500个example，分为三个难度水平：easy，medium和hard，每个难度水平500道题目。 ProverQA数据集由ProverGen框架生成，详见ICLR2025论文 Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation。该框架首次组合了LLMs和symbolic prover来生成逻辑推理数据，兼顾了生成数据的多样性和真实性。

The ProverQA dataset is primarily designed for evaluating the logical reasoning capabilities of Large Language Models (LLMs). It consists of a training set and an evaluation set, with 5000 examples in the training set and 1500 examples in the evaluation set. The dataset is divided into three difficulty levels: easy, medium, and hard, each containing 500 questions. The ProverQA dataset is generated using the ProverGen framework, which is detailed in the ICLR 2025 paper *Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation*. This framework is the first to combine LLMs and symbolic provers to generate logical reasoning data, balancing both the diversity and authenticity of the generated data.

搜集汇总

背景与挑战

背景概述

ProverQA是一个专注于评测大语言模型逻辑推理能力的数据集，包含6500个example，分为三个难度水平。数据集通过ProverGen框架生成，结合了LLMs和symbolic prover，确保数据的多样性和真实性。