---
license: mit
language:
- en
pretty_name: InfoBench
size_categories:
- n<1K
---
# Dataset Card for InFoBench Dataset
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Usage](#dataset-usage)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Additional Information](#additional-information)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
## Dataset Description
- **Repository:** [InFoBench Repository](https://github.com/qinyiwei/InfoBench)
- **Paper:** [InFoBench: Evaluating Instruction Following Ability in Large Language Models](https://arxiv.org/pdf/2401.03601.pdf)
The InFoBench Dataset is an evaluation benchmark dataset containing 500 instructions and corresponding 2250 decomposed requirements.
## Dataset Usage
You can directly download it with huggingface datasets.
``` python
from datasets import load_dataset
dataset = load_dataset("kqsong/InFoBench")
```
## Dataset Structure
### Data Instances
For each instance, there is an instruction string, an input string (optional), a list of decomposed questions, and a list of the labels for each decomposed question.
```json
{
"id": "domain_oriented_task_215",
"input": "",
"category": "Business and Economics: Business Administration",
"instruction": "Generate a non-disclosure agreement of two pages (each page is limited to 250 words) for a software development project involving Party A and Party B. The confidentiality duration should be 5 years. \n\nThe first page should include definitions for key terms such as 'confidential information', 'disclosure', and 'recipient'. \n\nOn the second page, provide clauses detailing the protocol for the return or destruction of confidential information, exceptions to maintaining confidentiality, and the repercussions following a breach of the agreement. \n\nPlease indicate the separation between the first and second pages with a full line of dashed lines ('-----'). Also, make sure that each page is clearly labeled with its respective page number.",
"decomposed_questions": [
"Is the generated text a non-disclosure agreement?",
"Does the generated text consist of two pages?",
"Is each page of the generated text limited to 250 words?",
"Is the generated non-disclosure agreement for a software development project involving Party A and Party B?",
"Does the generated non-disclosure agreement specify a confidentiality duration of 5 years?",
"Does the first page of the generated non-disclosure agreement include definitions for key terms such as 'confidential information', 'disclosure', and 'recipient'?",
"Does the second page of the generated non-disclosure agreement provide clauses detailing the protocol for the return or destruction of confidential information?",
"Does the second page of the generated non-disclosure agreement provide exceptions to maintaining confidentiality?",
"Does the second page of the generated non-disclosure agreement provide the repercussions following a breach of the agreement?",
"Does the generated text indicate the separation between the first and second pages with a full line of dashed lines ('-----')?",
"Does the generated text ensure that each page is clearly labeled with its respective page number?"
],
"subset": "Hard_set",
"question_label": [
["Format"],
["Format", "Number"],
["Number"],
["Content"],
["Content"],
["Format", "Content"],
["Content"],
["Content"],
["Content"],
["Format"],
["Format"]
]
}
```
### Data Fields
- `id`: a string.
- `subset`: `Hard_Set` or `Easy_Set`.
- `category`: a string containing categorical information.
- `instruction`: a string containing instructions.
- `input`: a string, containing the context information, could be an empty string.
- `decomposed_questions`: a list of strings, each corresponding to a decomposed requirement.
- `question_label`: a list of list of strings, each list of strings containing a series of labels for the corresponding decomposed questions.
## Additional Information
### Licensing Information
The InFoBench Dataset version 1.0.0 is released under the [MIT LISENCE](https://github.com/qinyiwei/InfoBench/blob/main/LICENSE)
### Citation Information
```
@article{qin2024infobench,
title={InFoBench: Evaluating Instruction Following Ability in Large Language Models},
author={Yiwei Qin and Kaiqiang Song and Yebowen Hu and Wenlin Yao and Sangwoo Cho and Xiaoyang Wang and Xuansheng Wu and Fei Liu and Pengfei Liu and Dong Yu},
year={2024},
eprint={2401.03601},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
许可证：MIT许可证
语言：
- 英语
友好名称：InfoBench
样本规模类别：
- 样本量小于1000
---
# InfoBench数据集卡片
## 目录
- [数据集描述](#dataset-description)
- [数据集使用](#dataset-usage)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [附加信息](#additional-information)
- [许可证信息](#licensing-information)
- [引用信息](#citation-information)
## 数据集描述
- **仓库地址：** [InfoBench仓库](https://github.com/qinyiwei/InfoBench)
- **论文链接：** [InFoBench：评估大语言模型（Large Language Model）的指令遵循能力](https://arxiv.org/pdf/2401.03601.pdf)
InfoBench数据集是一款评估基准数据集，包含500条指令以及对应的2250条分解需求。
## 数据集使用
可通过Hugging Face Datasets直接下载该数据集。
python
from datasets import load_dataset
dataset = load_dataset("kqsong/InFoBench")
## 数据集结构
### 数据实例
每个数据实例包含一条指令字符串、一条可选的输入字符串、一个分解问题列表，以及各分解问题对应的标签列表。
json
{
"id": "domain_oriented_task_215",
"input": "",
"category": "Business and Economics: Business Administration",
"instruction": "Generate a non-disclosure agreement of two pages (each page is limited to 250 words) for a software development project involving Party A and Party B. The confidentiality duration should be 5 years.
The first page should include definitions for key terms such as 'confidential information', 'disclosure', and 'recipient'.
On the second page, provide clauses detailing the protocol for the return or destruction of confidential information, exceptions to maintaining confidentiality, and the repercussions following a breach of the agreement.
Please indicate the separation between the first and second pages with a full line of dashed lines ('-----'). Also, make sure that each page is clearly labeled with its respective page number.",
"decomposed_questions": [
"Is the generated text a non-disclosure agreement?",
"Does the generated text consist of two pages?",
"Is each page of the generated text limited to 250 words?",
"Is the generated non-disclosure agreement for a software development project involving Party A and Party B?",
"Does the generated non-disclosure agreement specify a confidentiality duration of 5 years?",
"Does the first page of the generated non-disclosure agreement include definitions for key terms such as 'confidential information', 'disclosure', and 'recipient'?",
"Does the second page of the generated non-disclosure agreement provide clauses detailing the protocol for the return or destruction of confidential information?",
"Does the second page of the generated non-disclosure agreement provide exceptions to maintaining confidentiality?",
"Does the second page of the generated non-disclosure agreement provide the repercussions following a breach of the agreement?",
"Does the generated text indicate the separation between the first and second pages with a full line of dashed lines ('-----')?",
"Does the generated text ensure that each page is clearly labeled with its respective page number?"
],
"subset": "Hard_set",
"question_label": [
["Format"],
["Format", "Number"],
["Number"],
["Content"],
["Content"],
["Format", "Content"],
["Content"],
["Content"],
["Content"],
["Format"],
["Format"]
]
}
### 数据字段
- `id`：字符串类型。
- `subset`：取值为`Hard_Set`或`Easy_Set`。
- `category`：包含分类信息的字符串。
- `instruction`：包含指令内容的字符串。
- `input`：字符串类型，承载上下文信息，可为空字符串。
- `decomposed_questions`：字符串列表，每个元素对应一条分解需求。
- `question_label`：列表的列表，每个子列表包含对应分解问题的一系列标签。
## 附加信息
### 许可证信息
InfoBench数据集1.0.0版本依据[MIT许可证](https://github.com/qinyiwei/InfoBench/blob/main/LICENSE)发布。
### 引用信息
@article{qin2024infobench,
title={InFoBench: Evaluating Instruction Following Ability in Large Language Models},
author={Yiwei Qin and Kaiqiang Song and Yebowen Hu and Wenlin Yao and Sangwoo Cho and Xiaoyang Wang and Xuansheng Wu and Fei Liu and Pengfei Liu and Dong Yu},
year={2024},
eprint={2401.03601},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
