czy6/big_bench_hard

---
language:
- en
license: mit
size_categories:
- 1K<n<10K
task_categories:
- question-answering
- text2text-generation
- multiple-choice
- text-generation
tags:
- mathematical-reasoning
- geometry
pretty_name: BIG-Bench Hard
library_name: datasets
dataset_info:
- config_name: boolean_expressions
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: boolean_expressions
num_bytes: 11790
num_examples: 250
download_size: 4531
dataset_size: 11790
- config_name: causal_judgement
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: causal_judgement
num_bytes: 198021
num_examples: 187
download_size: 67829
dataset_size: 198021
- config_name: date_understanding
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: date_understanding
num_bytes: 61226
num_examples: 250
download_size: 17179
dataset_size: 61226
- config_name: disambiguation_qa
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: disambiguation_qa
num_bytes: 81635
num_examples: 250
download_size: 14551
dataset_size: 81635
- config_name: dyck_languages
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: dyck_languages
num_bytes: 38432
num_examples: 250
download_size: 9656
dataset_size: 38432
- config_name: few_shot_prompts
features:
- name: dataset_name
dtype: string
- name: answer_only_prompt
dtype: string
- name: chain_of_thought_prompt
dtype: string
splits:
- name: few_shot_prompts
num_bytes: 106260
num_examples: 27
download_size: 54761
dataset_size: 106260
- config_name: formal_fallacies
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: formal_fallacies
num_bytes: 138357
num_examples: 250
download_size: 34568
dataset_size: 138357
- config_name: geometric_shapes
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: geometric_shapes
num_bytes: 80560
num_examples: 250
download_size: 18585
dataset_size: 80560
- config_name: hyperbaton
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: hyperbaton
num_bytes: 40324
num_examples: 250
download_size: 10298
dataset_size: 40324
- config_name: logical_deduction_five_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: logical_deduction_five_objects
num_bytes: 154095
num_examples: 250
download_size: 15293
dataset_size: 154095
- config_name: logical_deduction_seven_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: logical_deduction_seven_objects
num_bytes: 199022
num_examples: 250
download_size: 19731
dataset_size: 199022
- config_name: logical_deduction_three_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: logical_deduction_three_objects
num_bytes: 108831
num_examples: 250
download_size: 10908
dataset_size: 108831
- config_name: movie_recommendation
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: movie_recommendation
num_bytes: 55332
num_examples: 250
download_size: 22579
dataset_size: 55332
- config_name: multistep_arithmetic_two
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: multistep_arithmetic_two
num_bytes: 12943
num_examples: 250
download_size: 7367
dataset_size: 12943
- config_name: navigate
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: navigate
num_bytes: 49031
num_examples: 250
download_size: 9544
dataset_size: 49031
- config_name: object_counting
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: object_counting
num_bytes: 30508
num_examples: 250
download_size: 10236
dataset_size: 30508
- config_name: penguins_in_a_table
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: penguins_in_a_table
num_bytes: 73269
num_examples: 146
download_size: 9719
dataset_size: 73269
- config_name: reasoning_about_colored_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: reasoning_about_colored_objects
num_bytes: 106244
num_examples: 250
download_size: 18723
dataset_size: 106244
- config_name: ruin_names
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: ruin_names
num_bytes: 50836
num_examples: 250
download_size: 18196
dataset_size: 50836
- config_name: salient_translation_error_detection
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: salient_translation_error_detection
num_bytes: 283860
num_examples: 250
download_size: 52804
dataset_size: 283860
- config_name: snarks
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: snarks
num_bytes: 39464
num_examples: 178
download_size: 16485
dataset_size: 39464
- config_name: sports_understanding
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: sports_understanding
num_bytes: 22723
num_examples: 250
download_size: 7917
dataset_size: 22723
- config_name: temporal_sequences
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: temporal_sequences
num_bytes: 143796
num_examples: 250
download_size: 30452
dataset_size: 143796
- config_name: tracking_shuffled_objects_five_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: tracking_shuffled_objects_five_objects
num_bytes: 168090
num_examples: 250
download_size: 31833
dataset_size: 168090
- config_name: tracking_shuffled_objects_seven_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: tracking_shuffled_objects_seven_objects
num_bytes: 215274
num_examples: 250
download_size: 41704
dataset_size: 215274
- config_name: tracking_shuffled_objects_three_objects
features:
- name: question
dtype: string
- name: choices
struct:
- name: label
list: string
- name: text
list: string
- name: target
dtype: string
splits:
- name: tracking_shuffled_objects_three_objects
num_bytes: 125104
num_examples: 250
download_size: 21772
dataset_size: 125104
- config_name: web_of_lies
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: web_of_lies
num_bytes: 45082
num_examples: 250
download_size: 14984
dataset_size: 45082
- config_name: word_sorting
features:
- name: question
dtype: string
- name: target
dtype: string
splits:
- name: word_sorting
num_bytes: 60918
num_examples: 250
download_size: 44118
dataset_size: 60918
configs:
- config_name: boolean_expressions
data_files:
- split: boolean_expressions
path: boolean_expressions/boolean_expressions-*
- config_name: causal_judgement
data_files:
- split: causal_judgement
path: causal_judgement/causal_judgement-*
- config_name: date_understanding
data_files:
- split: date_understanding
path: date_understanding/date_understanding-*
- config_name: disambiguation_qa
data_files:
- split: disambiguation_qa
path: disambiguation_qa/disambiguation_qa-*
- config_name: dyck_languages
data_files:
- split: dyck_languages
path: dyck_languages/dyck_languages-*
- config_name: few_shot_prompts
data_files:
- split: few_shot_prompts
path: few_shot_prompts/few_shot_prompts-*
- config_name: formal_fallacies
data_files:
- split: formal_fallacies
path: formal_fallacies/formal_fallacies-*
- config_name: geometric_shapes
data_files:
- split: geometric_shapes
path: geometric_shapes/geometric_shapes-*
- config_name: hyperbaton
data_files:
- split: hyperbaton
path: hyperbaton/hyperbaton-*
- config_name: logical_deduction_five_objects
data_files:
- split: logical_deduction_five_objects
path: logical_deduction_five_objects/logical_deduction_five_objects-*
- config_name: logical_deduction_seven_objects
data_files:
- split: logical_deduction_seven_objects
path: logical_deduction_seven_objects/logical_deduction_seven_objects-*
- config_name: logical_deduction_three_objects
data_files:
- split: logical_deduction_three_objects
path: logical_deduction_three_objects/logical_deduction_three_objects-*
- config_name: movie_recommendation
data_files:
- split: movie_recommendation
path: movie_recommendation/movie_recommendation-*
- config_name: multistep_arithmetic_two
data_files:
- split: multistep_arithmetic_two
path: multistep_arithmetic_two/multistep_arithmetic_two-*
- config_name: navigate
data_files:
- split: navigate
path: navigate/navigate-*
- config_name: object_counting
data_files:
- split: object_counting
path: object_counting/object_counting-*
- config_name: penguins_in_a_table
data_files:
- split: penguins_in_a_table
path: penguins_in_a_table/penguins_in_a_table-*
- config_name: reasoning_about_colored_objects
data_files:
- split: reasoning_about_colored_objects
path: reasoning_about_colored_objects/reasoning_about_colored_objects-*
- config_name: ruin_names
data_files:
- split: ruin_names
path: ruin_names/ruin_names-*
- config_name: salient_translation_error_detection
data_files:
- split: salient_translation_error_detection
path: salient_translation_error_detection/salient_translation_error_detection-*
- config_name: snarks
data_files:
- split: snarks
path: snarks/snarks-*
- config_name: sports_understanding
data_files:
- split: sports_understanding
path: sports_understanding/sports_understanding-*
- config_name: temporal_sequences
data_files:
- split: temporal_sequences
path: temporal_sequences/temporal_sequences-*
- config_name: tracking_shuffled_objects_five_objects
data_files:
- split: tracking_shuffled_objects_five_objects
path: tracking_shuffled_objects_five_objects/tracking_shuffled_objects_five_objects-*
- config_name: tracking_shuffled_objects_seven_objects
data_files:
- split: tracking_shuffled_objects_seven_objects
path: tracking_shuffled_objects_seven_objects/tracking_shuffled_objects_seven_objects-*
- config_name: tracking_shuffled_objects_three_objects
data_files:
- split: tracking_shuffled_objects_three_objects
path: tracking_shuffled_objects_three_objects/tracking_shuffled_objects_three_objects-*
- config_name: web_of_lies
data_files:
- split: web_of_lies
path: web_of_lies/web_of_lies-*
- config_name: word_sorting
data_files:
- split: word_sorting
path: word_sorting/word_sorting-*
---

All rights and obligations of the dataset are with original authors of the paper/dataset.
I have merely made this dataset with a MIT licence available on HuggingFace.

# BIG-Bench Hard Dataset

This repository contains a copy of the [BIG-Bench Hard](https://arxiv.org/abs/2210.09261) dataset.
Small edits to the formatting of the dataset are made to integrate it into the [Inspect Evals](https://ukgovernmentbeis.github.io/inspect_evals/) repository, a community contributed LLM
evaulations for [Inspect AI](https://inspect.ai-safety-institute.org.uk/) a framework by the [UK AI Safety Institute](https://www.aisi.gov.uk/).

The BIG-Bench Hard dataset is a collection of various task categories, with each task focused on testing specific reasoning, logic, or language abilities.
The dataset also includes two types of 3-shot prompts for each task: answer-only prompts and chain-of-thought prompts.

## Dataset Structure

### Main Task Datasets

The collection includes a wide range of tasks, with each designed to evaluate different aspects of logical reasoning, understanding, and problem-solving abilities. Below is a list of all included tasks:

1. **Boolean Expressions**
- Evaluate the truth value of a Boolean expression using Boolean constants (`True`, `False`) and basic operators (`and`, `or`, `not`).

2. **Causal Judgment**
- Given a short story, determine the likely answer to a causal question about the story based on moral, intentional, or counterfactual analysis.

3. **Date Understanding**
- Manipulate and reason about dates in various formats by converting date formats, calculating intervals, and answering related questions.

4. **Disambiguation QA**
- Resolve ambiguous pronouns or determine if a pronoun’s reference is inherently ambiguous, identifying the correct antecedent where possible.

5. **Dyck Languages**
- Predict the sequence of closing parentheses for a Dyck-4 word sequence, given an incomplete set of parentheses.

6. **Formal Fallacies Syllogisms Negation**
- Assess logical validity in informal arguments, with a focus on understanding deductive validity versus formal fallacies involving negations.

7. **Geometric Shapes**
- Given an SVG path with multiple commands, determine the resulting geometric shape.

8. **Hyperbaton (Adjective Ordering)**
- Determine the grammatically correct sentence from two English sentences with different adjective orders.

9. **Logical Deduction**
- Deduce the order of a sequence of objects based on clues about spatial relationships and placements.

10. **Movie Recommendation**
- Recommend a new movie based on a user's viewing history from four potential choices.

11. **Multi-Step Arithmetic**
- Solve multi-step arithmetic equations involving basic operations like addition, subtraction, multiplication, and division.

12. **Navigate**
- Predict whether an agent will return to its starting point after a series of navigation steps.

13. **Object Counting**
- Given a list of possessions with quantities, determine the total count of a specific object class (e.g., fruits).

14. **Penguins in a Table**
- Answer attribute-related questions about penguins based on a unique table format, sometimes with additional context.

15. **Reasoning about Colored Objects**
- Answer questions about the color of objects based on contextual information.

16. **Ruin Names**
- Identify a humorous one-character edit to the name of an artist, band, or movie.

17. **Salient Translation Error Detection**
- Determine the type of error in the English translation of a German source sentence.

18. **Snarks**
- Distinguish between two nearly-identical sentences to identify which one is sarcastic.

19. **Sports Understanding**
- Judge whether a factitious sentence about sports is plausible.

20. **Temporal Sequences**
- Based on a series of daily activities, determine when the person might have been free for another activity.

21. **Tracking Shuffled Objects**
- Track the final positions of objects after a series of pairwise swaps from an initial arrangement.

22. **Web of Lies**
- Evaluate the truth value of a Boolean function expressed as a natural-language word problem.

23. **Word Sorting**
- Sort a list of words lexicographically.

Each dataset contains:
- `question`: The task question text
- `choices`: Multiple choice options
- `label`: List of choice identifiers (A, B, C, etc.)
- `text`: List of choice texts
- `target`: Correct answer label

### Few-Shot Prompts

The `few_shot_prompts` dataset provides example prompts for each task type with two formats:
- `answer_only_prompt`: Direct answer template
- `chain_of_thought_prompt`: Template encouraging step-by-step reasoning

## Usage

### Loading the Dataset

```python
from datasets import load_dataset

# Load a specific task
date_dataset = load_dataset("Joschka/big_bench_hard", "date_understanding")

# Load prompts
prompts = load_dataset("Joschka/big_bench_hard", "few_shot_prompts")
```

### Using Few-Shot Prompts

```python
def get_task_prompts(prompts_dataset, task_name):
prompt_data = prompts_dataset['few_shot_prompts'].filter(
lambda x: x['dataset_name'] == task_name
)[0]
return {
'answer_only': prompt_data['answer_only_prompt'],
'chain_of_thought': prompt_data['chain_of_thought_prompt']
}

# Get prompts for date understanding task
date_prompts = get_task_prompts(prompts, 'date_understanding')
```

## Data Files

Each dataset configuration includes its own data files:
- `boolean_expressions/boolean_expressions-*`
- `causal_judgment/causal_judgment-*`
- `date_understanding/date_understanding-*`
- `disambiguation_qa/disambiguation_qa-*`
- `dyck_languages/dyck_languages-*`
- `formal_fallacies_syllogisms_negation/formal_fallacies_syllogisms_negation-*`
- `geometric_shapes/geometric_shapes-*`
- `hyperbaton/hyperbaton-*`
- `logical_deduction/logical_deduction-*`
- `movie_recommendation/movie_recommendation-*`
- `multi_step_arithmetic/multi_step_arithmetic-*`
- `navigate/navigate-*`
- `object_counting/object_counting-*`
- `penguins_in_a_table/penguins_in_a_table-*`
- `reasoning_about_colored_objects/reasoning_about_colored_objects-*`
- `ruin_names/ruin_names-*`
- `salient_translation_error_detection/salient_translation_error_detection-*`
- `snarks/snarks-*`
- `sports_understanding/sports_understanding-*`
- `temporal_sequences/temporal_sequences-*`
- `tracking_shuffled_objects_five_objects/tracking_shuffled_objects_five_objects-*`
- `tracking_shuffled_objects_seven_objects`
- `tracking_shuffled_objects_three_objects/tracking_shuffled_objects_three_objects-*`
- `web_of_lies/web_of_lies-*`
- `word_sorting/word_sorting-*`

## Citation
If your research makes use of this dataset please cite the BIG-Bench Hard paper.

**BIG-Bench Hard** ([_Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them_ (Suzgun et al., 2022)](https://arxiv.org/abs/2210.09261))
```
@article{suzgun2022challenging,
title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
author={Suzgun, Mirac and Scales, Nathan and Sch{\"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and and Wei, Jason},
journal={arXiv preprint arXiv:2210.09261},
year={2022}
}
```

[BIG-Bench Hard](https://arxiv.org/abs/2210.09261), focuss on a suite of 23 challenging BIG-Bench tasks which we call **BIG-Bench Hard (BBH)**. These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average humanrater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.

**BIG Bench** ([_Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models_ (Srivastava et al., 2022)](https://arxiv.org/abs/2206.04615))
```
@article{srivastava2022beyond,
title={Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models},
author={Srivastava, Aarohi and Rastogi, Abhinav and Rao, Abhishek and Shoeb, Abu Awal Md and Abid, Abubakar and Fisch, Adam and Brown, Adam R and Santoro, Adam and Gupta, Aditya and Garriga-Alonso, Adri{\`a} and others},
journal={arXiv preprint arXiv:2206.04615},
year={2022}
}
```

## Related Research
This dataset is referenced in the survey paper [A Survey of Deep Learning for Geometry Problem Solving](https://huggingface.co/papers/2507.11936), where it is discussed in the context of geometry problem solving tasks, specifically its `geometric_shapes` subtask. The accompanying GitHub repository for this survey, which maintains a list of relevant papers and datasets, can be found here: [https://github.com/majianz/gps-survey](https://github.com/majianz/gps-survey).

### Abstract
Geometry problem solving is a key area of mathematical reasoning, which is widely involved in many important fields such as education, mathematical ability assessment of artificial intelligence, and multimodal ability assessment. In recent years, the rapid development of deep learning technology, especially the rise of multimodal large language models, has triggered a widespread research boom. This paper provides a survey of the applications of deep learning in geometry problem solving, including (i) a comprehensive summary of the relevant tasks in geometry problem solving; (ii) a thorough review of related deep learning methods; (iii) a detailed analysis of evaluation metrics and methods; and (iv) a critical discussion of the current challenges and future directions that can be explored. Our goal is to provide a comprehensive and practical reference of deep learning for geometry problem solving to promote further developments in this field. We create a continuously updated list of papers on GitHub: this https URL .

## Abstract
[BIG-Bench](https://github.com/google/BIG-bench) ([Srivastava et al., 2022](https://arxiv.org/abs/2206.04615)) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models?

## License

This dataset is licensed under MIT.

本数据集元信息如下：
- 语言：英语
- 许可证：MIT
- 样本规模：1000 < 样本数量 < 10000
- 任务类别：问答、文本到文本生成、多项选择、文本生成
- 标签：数学推理、几何
- 友好名称：BIG-Bench Hard
- 依赖库：`datasets`

本数据集的所有权利与义务均归属于原论文/数据集的作者，本人仅将采用MIT许可证的本数据集部署至HuggingFace平台。

# BIG-Bench Hard 数据集

本仓库包含[BIG-Bench Hard（BBH）](https://arxiv.org/abs/2210.09261)数据集的复刻版本。为适配集成至[Inspect Evals](https://ukgovernmentbeis.github.io/inspect_evals/)仓库——该仓库是面向由英国人工智能安全研究所（UK AI Safety Institute）开发的[Inspect AI](https://inspect.ai-safety-institute.org.uk/)框架的社区共建大语言模型（LLM）评测集——我们对数据集格式进行了小幅调整。

BIG-Bench Hard数据集涵盖多类任务，每项任务均针对特定的推理、逻辑或语言能力展开评测。本数据集还为每项任务提供两类3样本提示（prompt）：仅输出答案提示与思维链（Chain-of-Thought, CoT）提示。

## 数据集结构

### 主任务数据集
本数据集包含一系列任务，旨在全面评测逻辑推理、理解与问题解决能力的多个维度。以下为全部收录任务列表：
1. **布尔表达式（Boolean Expressions）**：利用布尔常量（`True`、`False`）与基础逻辑运算符（`and`、`or`、`not`）判断布尔表达式的真值。
2. **因果判断（Causal Judgment）**：给定短篇故事，基于道德、意向性或反事实分析，回答针对故事的因果类问题并给出合理结论。
3. **日期理解（Date Understanding）**：处理并推理多种格式的日期，包括转换日期格式、计算时间间隔并回答相关问题。
4. **消歧问答（Disambiguation QA）**：消解指代歧义或判断代词引用是否存在固有歧义，尽可能确定正确的先行词。
5. **Dyck语言（Dyck Languages）**：给定不完整的括号序列，为Dyck-4语言序列预测闭合括号的正确顺序。
6. **形式谬误（Formal Fallacies Syllogisms Negation）**：评估非形式论证的逻辑有效性，重点区分演绎有效性与涉及否定的形式谬误。
7. **几何形状（Geometric Shapes）**：给定包含多类命令的可缩放矢量图形（SVG）路径，推导对应的几何形状。
8. **语序错乱（Hyperbaton，形容词排序）**：从两个形容词语序不同的英语句子中，选出语法正确的句子。
9. **逻辑演绎（Logical Deduction）**：基于关于空间关系与位置的线索，推导一系列对象的排列顺序。
10. **电影推荐（Movie Recommendation）**：基于用户的观影历史，从四个候选选项中推荐合适的新电影。
11. **多步算术（Multi-Step Arithmetic）**：求解包含加减乘除等基础运算的多步算术方程。
12. **路径导航（Navigate）**：预测智能体在执行一系列导航步骤后是否能返回起点。
13. **物体计数（Object Counting）**：给定带有数量的物品列表，计算特定类别物品的总数量（例如水果）。
14. **桌上面的企鹅（Penguins in a Table）**：基于特定表格格式的信息（有时会附带额外上下文），回答关于企鹅的属性相关问题。
15. **有色物体推理（Reasoning about Colored Objects）**：基于上下文信息，回答关于物体颜色的相关问题。
16. **名称恶搞（Ruin Names）**：识别对艺术家、乐队或电影名称进行的单字符幽默修改。
17. **显著翻译错误检测（Salient Translation Error Detection）**：判断德语源句的英语译文中存在的错误类型。
18. **讽刺识别（Snarks）**：区分两句几乎完全相同的句子，识别其中哪一句带有讽刺意味。
19. **体育理解（Sports Understanding）**：判断一则关于体育的虚构句子是否符合常理。
20. **时间序列（Temporal Sequences）**：基于一系列日常活动，推断人物何时有空参与另一项活动。
21. **跟踪置换物体（Tracking Shuffled Objects）**：从初始排列开始，跟踪物体经过一系列两两交换后的最终位置。
22. **谎言网络（Web of Lies）**：将自然语言描述的布尔函数转换为自然语言问题，评估其真值。
23. **单词排序（Word Sorting）**：按照字典顺序对单词列表进行排序。

每个数据集均包含以下字段：
- `question`：任务问题文本
- `choices`：多项选择选项
- `label`：选项标识符列表（如A、B、C等）
- `text`：选项文本列表
- `target`：正确答案标签

### 少样本提示
`few_shot_prompts`数据集为各类任务提供两种格式的示例提示：
- `answer_only_prompt`：仅要求输出答案的提示模板
- `chain_of_thought_prompt`：鼓励逐步推理的提示模板

## 使用方法

### 加载数据集
python
from datasets import load_dataset

# 加载特定任务
date_dataset = load_dataset("Joschka/big_bench_hard", "date_understanding")

# 加载提示集
prompts = load_dataset("Joschka/big_bench_hard", "few_shot_prompts")

### 使用少样本提示
python
def get_task_prompts(prompts_dataset, task_name):
prompt_data = prompts_dataset['few_shot_prompts'].filter(
lambda x: x['dataset_name'] == task_name
)[0]
return {
'answer_only': prompt_data['answer_only_prompt'],
'chain_of_thought': prompt_data['chain_of_thought_prompt']
}

# 获取日期理解任务的提示
date_prompts = get_task_prompts(prompts, 'date_understanding')

## 数据文件
每个数据集配置均包含独立的数据文件：
- `boolean_expressions/boolean_expressions-*`
- `causal_judgement/causal_judgement-*`
- `date_understanding/date_understanding-*`
- `disambiguation_qa/disambiguation_qa-*`
- `dyck_languages/dyck_languages-*`
- `formal_fallacies_syllogisms_negation/formal_fallacies_syllogisms_negation-*`
- `geometric_shapes/geometric_shapes-*`
- `hyperbaton/hyperbaton-*`
- `logical_deduction/logical_deduction-*`
- `movie_recommendation/movie_recommendation-*`
- `multi_step_arithmetic/multi_step_arithmetic-*`
- `navigate/navigate-*`
- `object_counting/object_counting-*`
- `penguins_in_a_table/penguins_in_a_table-*`
- `reasoning_about_colored_objects/reasoning_about_colored_objects-*`
- `ruin_names/ruin_names-*`
- `salient_translation_error_detection/salient_translation_error_detection-*`
- `snarks/snarks-*`
- `sports_understanding/sports_understanding-*`
- `temporal_sequences/temporal_sequences-*`
- `tracking_shuffled_objects_five_objects/tracking_shuffled_objects_five_objects-*`
- `tracking_shuffled_objects_seven_objects/tracking_shuffled_objects_seven_objects-*`
- `tracking_shuffled_objects_three_objects/tracking_shuffled_objects_three_objects-*`
- `web_of_lies/web_of_lies-*`
- `word_sorting/word_sorting-*`

## 引用说明
若您的研究使用本数据集，请引用BIG-Bench Hard论文。

**BIG-Bench Hard**（[《挑战性BIG-Bench任务与思维链能否解决它们》（Suzgun等，2022）](https://arxiv.org/abs/2210.09261)）
bibtex
@article{suzgun2022challenging,
title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
author={Suzgun, Mirac and Scales, Nathan and Schärli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and Wei, Jason},
journal={arXiv preprint arXiv:2210.09261},
year={2022}
}

BIG-Bench Hard（BBH）聚焦于23项具有挑战性的BIG-Bench任务集合。这些任务是此前大语言模型评测未达到普通人类评分者平均水平的任务。我们发现，对BBH任务应用思维链（CoT）提示，可使PaLM在23项任务中的10项上超越普通人类评分者的表现，而Codex（code-davinci-002）则可在23项任务中的17项上达成该目标。由于BBH中的许多任务都需要多步推理，如BIG-Bench评测（Srivastava等，2022）中所做的那样，不使用CoT的少样本提示会大幅低估语言模型的最佳性能与能力，而通过CoT提示可更好地捕捉这些能力。进一步的分析探索了CoT与模型规模在BBH上的交互作用，结果表明CoT可在多项原本性能随模型规模增长平缓的BBH任务上实现涌现式任务性能。

**BIG Bench**（[《超越模仿游戏：量化与外推大语言模型的能力》（Srivastava等，2022）](https://arxiv.org/abs/2206.04615)）
bibtex
@article{srivastava2022beyond,
title={Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models},
author={Srivastava, Aarohi and Rastogi, Abhinav and Rao, Abhishek and Shoeb, Abu Awal Md and Abid, Abubakar and Fisch, Adam and Brown, Adam R and Santoro, Adam and Gupta, Aditya and Garriga-Alonso, Adrià and others},
journal={arXiv preprint arXiv:2206.04615},
year={2022}
}

## 相关研究
本数据集被综述论文《深度学习在几何问题求解中的应用》[https://huggingface.co/papers/2507.11936](https://huggingface.co/papers/2507.11936)引用，该论文从几何问题求解任务的角度讨论了本数据集，特别是其`geometric_shapes`子任务。该综述附带的GitHub仓库维护了相关论文与数据集的列表，地址为：[https://github.com/majianz/gps-survey](https://github.com/majianz/gps-survey)。

### 摘要
几何问题求解是数学推理的关键领域，广泛涉及教育、人工智能数学能力评估、多模态能力评估等多个重要领域。近年来，深度学习技术的快速发展，尤其是多模态大语言模型的兴起，引发了广泛的研究热潮。本文综述了深度学习在几何问题求解中的应用，包括：（i）全面总结几何问题求解中的相关任务；（ii）深入回顾相关深度学习方法；（iii）详细分析评测指标与方法；（iv）批判性讨论当前面临的挑战与可探索的未来方向。我们的目标是为几何问题求解领域的深度学习应用提供全面且实用的参考，以推动该领域的进一步发展。我们在GitHub上创建了持续更新的论文列表：该链接。

另一段摘要：
[BIG-Bench](https://github.com/google/BIG-bench)（Srivastava等，2022）是一个多样化的评测套件，聚焦于被认为超出当前大语言模型能力范围的任务。语言模型在该基准上已取得不错的进展，BIG-Bench论文中表现最佳的模型通过少样本提示，在65%的BIG-Bench任务上超越了已报道的人类评分者平均结果。但语言模型在哪些任务上未达到人类评分者的平均水平，而这些任务是否真的无法被当前的语言模型解决？

## 许可证
本数据集采用MIT许可证进行授权。