ibm-research/SQL-API-Bench

SQL-API-Bench数据集是一个包含问答任务的数据集，需要同时访问数据库和API。该数据集由两个新的基准组成，这两个基准包含的问题需要结合数据库和API调用才能得到答案。Benchmark I通过替换Spider数据库表的一部分为通过API执行的等效表，来测试数据库和API调用组合的机制，而不需要改变原始Spider基准的问题或它们的真实答案。Benchmark II引入了一组新的标量API，执行简单的词汇、数值或地理空间操作。从Spider数据库的二十多个子集中，我们将原始Spider数据库中的问题转化为需要交织数据库操作和1-3个标量API组合的新问题。通过半自动化过程，我们建立了一组相应的真实答案，生成了2300多个人工审核的问题/答案对。

The SQL-API-Bench dataset is a QA dataset that requires simultaneous access to databases and APIs. The dataset consists of two new benchmarks that contain questions which require a combination of database and API calls to answer. Benchmark I replaces a fraction of the actual Spider database tables with equivalents that are executed via APIs, allowing us to test the mechanism by which database and API calls are combined without having to change the questions or their ground-truth answers from the original Spider benchmark. Benchmark II introduces a new set of scalar APIs that perform simple lexical, numeric, or geo-spatial operations. From a subset of two dozen Spider databases, we transform questions from the original Spider database into new questions that require interleaving database operations with compositions of 1-3 scalar APIs. We establish a set of corresponding ground-truth answers through a semi-automated process that generates over 2300 human-vetted question/answer pairs.