SQL-API-Bench

<!--
# My Great Model
This is the main content of your model card... -->

# Dataset Card for Dataset Name

<!-- Provide a quick summary of the dataset. -->
This dataset contains QA that requires DB and API access at the same time. It is composed of two new benchmarks consisting of questions whose answers require a
combination of database and API calls, both of
which are augmentations of the popular Spider
dataset and benchmark.

Benchmark I replaces a fraction of the real Spider database tables with
equivalents that are executed via APIs. This allows us to directly test the mechanism by which
database and API calls are combined without having to change the questions or their ground-truth answers from the original Spider benchmark.

Benchmark II introduces a new set of scalar APIs that perform simple lexical, numeric, or geo-spatial operations. From a subset of two dozen Spider
databases, we transform questions from the original Spider database into new questions that
require interleaving database operations with compositions of 1-3 scalar APIs. We establish a set
of corresponding ground-truth answers through a
semi-automated process that generates over 2300
human-vetted question/answer pairs.

<!-- This dataset card aims to be a base template for new datasets. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1). -->

<!-- ## Dataset Details
This dataset contains QA that requires DB and API access at the same time.
-->
<!-- ### Dataset Description -->

<!-- Provide a longer summary of what this dataset is. -->

<!-- - **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
-->
<!-- ### Dataset Sources [optional] -->

<!-- Provide the basic links for the dataset. -->

<!-- - **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
-->
<!-- ## Uses -->

<!-- Address questions around how the dataset is intended to be used. -->

<!-- ### Direct Use -->

<!-- This section describes suitable use cases for the dataset. -->

<!-- [More Information Needed] -->

<!-- ### Out-of-Scope Use -->

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

<!-- [More Information Needed] -->

<!-- ## Dataset Structure -->

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

<!-- [More Information Needed] -->

<!-- ## Dataset Creation -->

<!-- ### Curation Rationale -->

<!-- Motivation for the creation of this dataset. -->

<!-- [More Information Needed] -->

<!-- ### Source Data -->

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

<!-- #### Data Collection and Processing -->

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

<!-- [More Information Needed] -->

<!-- #### Who are the source data producers? -->

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

<!-- [More Information Needed] -->

<!-- ### Annotations [optional] -->

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

<!-- #### Annotation process -->

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

<!-- [More Information Needed] -->

<!-- #### Who are the annotators? -->

<!-- This section describes the people or systems who created the annotations. -->

<!-- [More Information Needed] -->

<!-- #### Personal and Sensitive Information -->

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

<!-- [More Information Needed] -->

<!-- ## Bias, Risks, and Limitations -->

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

<!-- [More Information Needed] -->

<!-- ### Recommendations -->

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

<!-- Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations. -->

<!-- ## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

<!-- **BibTeX:**

[More Information Needed]
**APA:**

[More Information Needed]

## Glossary [optional]
-->
<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

<!-- [More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed] -->

## Dataset Card Contact
[e-mail]
<!-- [More Information Needed] -->

# 数据集卡片（Dataset Card）：数据集名称

<!-- 请提供该数据集的简要概述。 -->
本数据集包含需同时调用数据库（Database，DB）与应用程序编程接口（Application Programming Interface，API）的问答（Question Answering，QA）样本。其由两项全新基准测试构成，所有测试问题的答案均需结合数据库操作与API调用完成，且两项基准测试均基于热门的Spider数据集与基准测试拓展而来。

基准测试I（Benchmark I）将部分真实的Spider数据库表替换为通过API执行的等效表。此举使我们能够直接测试数据库与API调用的结合机制，且无需修改原始Spider基准测试中的问题及其标准答案（ground-truth answers）。

基准测试II（Benchmark II）引入了一组全新的标量API（scalar APIs），可执行简单的词汇、数值或地理空间（geo-spatial）操作。我们从24个Spider数据库的子集中，将原始Spider数据库中的问题转换为新问题，这些新问题需要将数据库操作与1至3个标量API的组合调用交错进行。我们通过半自动化流程生成了超过2300个人工审核的问答对，并据此构建了对应的标准答案集。

<!-- 本数据集卡片旨在作为新数据集的基础模板，其基于[此原始模板](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1)生成。 -->

<!-- ## 数据集详情
本数据集包含需同时调用数据库与API的问答样本。
-->

<!-- ### 数据集概述 -->

<!-- 请提供该数据集的详细概述。 -->

<!-- - **整理者：** [需补充更多信息]
- **资助方[可选]：** [需补充更多信息]
- **分享方[可选]：** [需补充更多信息]
- **自然语言处理所用语言：** [需补充更多信息]
- **授权协议：** [需补充更多信息]
-->

<!-- ### 数据集来源[可选] -->

<!-- 请提供该数据集的基础链接。 -->

<!-- - **代码仓库：** [需补充更多信息]
- **相关论文[可选]：** [需补充更多信息]
- **演示示例[可选]：** [需补充更多信息]
-->

<!-- ## 用途 -->

<!-- 阐述该数据集的预期使用场景相关问题。 -->

<!-- ### 直接用途 -->

<!-- 本节描述该数据集的适用用例。 -->

<!-- [需补充更多信息] -->

<!-- ### 超出范围的用途 -->

<!-- 本节阐述误用、恶意使用以及该数据集无法良好适配的使用场景。 -->

<!-- [需补充更多信息] -->

<!-- ## 数据集结构 -->

<!-- 本节描述数据集的字段信息，以及其他与数据集结构相关的额外信息，例如划分数据集的标准、数据点间的关系等。 -->

<!-- [需补充更多信息] -->

<!-- ## 数据集创建 -->

<!-- ### 整理依据 -->

<!-- 创建该数据集的动机。 -->

<!-- [需补充更多信息] -->

<!-- ### 源数据 -->

<!-- 本节描述源数据（例如新闻文本与标题、社交媒体帖子、翻译后的语句等）。 -->

<!-- #### 数据收集与处理流程 -->

<!-- 本节描述数据收集与处理过程，例如数据选择标准、过滤与归一化方法、所用工具与库等。 -->

<!-- [需补充更多信息] -->

<!-- #### 源数据生产者是谁？ -->

<!-- 本节描述最初创建该数据的个人或系统。若可获取源数据创建者的自我报告人口统计或身份信息，也应在此处说明。 -->

<!-- [需补充更多信息] -->

<!-- ### 标注[可选] -->

<!-- 若数据集包含非初始数据收集阶段产生的标注，请用本节描述标注信息。 -->

<!-- #### 标注流程 -->

<!-- 本节描述标注流程，例如过程中使用的标注工具、标注数据量、提供给标注者的标注指南、标注者间统计数据、标注验证等。 -->

<!-- [需补充更多信息] -->

<!-- #### 标注者是谁？ -->

<!-- 本节描述创建标注的个人或系统。 -->

<!-- [需补充更多信息] -->

<!-- #### 个人与敏感信息 -->

<!-- 说明数据集是否包含可被视为个人、敏感或私密的数据（例如揭示地址、唯一可识别的姓名或别名、种族或族裔出身、性取向、宗教信仰、政治观点、财务或健康数据等）。若已对数据进行匿名化处理，请描述匿名化流程。 -->

<!-- [需补充更多信息] -->

<!-- ## 偏差、风险与局限性 -->

<!-- 本节旨在阐述技术与社会技术层面的局限性。 -->

<!-- ### 建议 -->

<!-- 本节旨在针对数据集的偏差、风险与技术局限性给出相关建议。 -->

<!-- 用户应知晓该数据集存在的风险、偏差与局限性。如需进一步建议，请补充更多信息。 -->

<!-- ## 引用[可选] -->

<!-- 若有介绍该数据集的论文或博客文章，应在此处附上APA与Bibtex格式的引用信息。 -->

<!-- **BibTeX：**

[需补充更多信息]
**APA格式：**

[需补充更多信息]

## 术语表[可选] -->

<!-- 若有需要，请在本节包含可帮助读者理解数据集或数据集卡片的术语与计算公式。 -->

<!-- [需补充更多信息]

## 更多信息[可选]

[需补充更多信息]

## 数据集卡片作者[可选]

[需补充更多信息] -->

## 数据集卡片联系人
[e-mail]
<!-- [需补充更多信息] -->