HiTZ/BertaQA

---
task_categories:
- question-answering
language:
- eu
- en
license:
- cc-by-4.0
pretty_name: BertaQA
size_categories:
- 1K<n<10K
configs:
- config_name: eu
data_files:
- split: test
path: "eustrivia_zuzenduta.jsonl"
- config_name: en
data_files:
- split: test
path: "eustrivia_elhuyar_zuzenduta.jsonl"
- config_name: en_mt_nllb
data_files:
- split: test
path: "eustrivia_nllb_zuzenduta.jsonl"
- config_name: en_mt_madlad
data_files:
- split: test
path: "eustrivia_madlad_zuzenduta.jsonl"
- config_name: en_mt_hitz
data_files:
- split: test
path: "eustrivia_hitz_zuzenduta.jsonl"
- config_name: en_mt_itzuli
data_files:
- split: test
path: "eustrivia_itzuli_zuzenduta.jsonl"
- config_name: en_mt_latxa-7b-v1.1
data_files:
- split: test
path: "eustrivia_latxa-7b-v1.1_zuzenduta.jsonl"
- config_name: en_mt_latxa-13b-v1.1
data_files:
- split: test
path: "eustrivia_latxa-13b-v1.1_zuzenduta.jsonl"
- config_name: en_mt_latxa-70b-v1.1
data_files:
- split: test
path: "eustrivia_latxa-70b-v1.1_zuzenduta.jsonl"
- config_name: en_mt_latxa-7b-v1
data_files:
- split: test
path: "eustrivia_latxa-7b-v1_zuzenduta.jsonl"
- config_name: en_mt_latxa-13b-v1
data_files:
- split: test
path: "eustrivia_latxa-13b-v1_zuzenduta.jsonl"
- config_name: en_mt_latxa-70b-v1
data_files:
- split: test
path: "eustrivia_latxa-70b-v1_zuzenduta.jsonl"
- config_name: en_mt_llama-2-7b
data_files:
- split: test
path: "eustrivia_llama-2-7b_zuzenduta.jsonl"
- config_name: en_mt_llama-2-13b
data_files:
- split: test
path: "eustrivia_llama-2-13b_zuzenduta.jsonl"
- config_name: en_mt_llama-2-70b
data_files:
- split: test
path: "eustrivia_llama-2-70b_zuzenduta.jsonl"
- config_name: en_mt_gemma-7b
data_files:
- split: test
path: "eustrivia_gemma-7b_zuzenduta.jsonl"
---
# Dataset Card for BertaQA

BertaQA is a trivia dataset comprising 4,756 multiple-choice trivia questions, with one single correct answer and 2 additional distractors. Crucially, questions are distributed between local and global topics. Whereas answering questions in the latter group requires general world knowledge, local questions require specific knowledge about the Basque Country and its culture. Additionally, questions are classified into eight categories, namely Basque and Literature, Geography and History, Society and Tradition, Sports and Leisure, Culture and Art, Music and Dance, Science and Technology, and Cinema and Shows. Questions also have three levels of difficulty: easy, medium or hard.

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

BertaQA is a trivia dataset comprising 4,756 multiple-choice questions, with a single correct answer and 2 additional distractors. Crucially, questions are distributed between *local* and *global* topics. Local questions require specific knowledge about the Basque Country and its culture, while global questions require more general world knowledge. Additionally, questions are classified into eight categories: Basque and Literature, Geography and History, Society and Traditions, Sports and Leisure, Culture and Art, Music and Dance, Science and Technology, and Cinema and Shows. Questions are also labeled according to their difficulty as easy, medium or hard.

The dataset was originally compiled in Basque by crawling public sources that are no longer available. Google does not return any result when searching for questions from the dataset verbatim. While this cannot categorically discard contamination, we believe that this, along with the nature of the raw data we crawled and the results from our experiments, makes it very unlikely that existing models were exposed to the same data during training.

Starting from the original version in Basque, we also created an English version of BertaQA using a professional translation service. Translators were instructed to use a consistent format for all the questions and answers, and we refined our guidelines through multiple rounds. For named entities, Wikipedia was used as a reference when available. During the translation process, a few of the original questions in Basque were corrected, either because the original answer was incorrect or it became outdated. In addition, we discarded a few questions that required knowledge of Basque or English, and would lose their essence if translated.

The resulting dataset is balanced regarding the number of questions per category and subset, with around 300 questions in each. The number of questions per difficulty is also balanced: most categories have around 110 easy and medium questions and 80 difficult questions in each subset. The average length of the questions and the candidates is around 50 and 13 characters, respectively.

- **Curated by:** HiTZ Center -- Ixa, University of the Basque Country (UPV/EHU)
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** Basque (eu), English (en)
- **License:** [More Information Needed]

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** https://github.com/juletx/BertaQA
- **Paper:** https://arxiv.org/abs/2406.07302
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the dataset is intended to be used. -->

### Direct Use

<!-- This section describes suitable use cases for the dataset. -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

[More Information Needed]

#### Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

[More Information Needed]

### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

#### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

[More Information Needed]

#### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

[More Information Needed]

#### Personal and Sensitive Information

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```bibtex
@misc{etxaniz2024bertaqa,
title={BertaQA: How Much Do Language Models Know About Local Culture?},
author={Julen Etxaniz and Gorka Azkune and Aitor Soroa and Oier Lopez de Lacalle and Mikel Artetxe},
year={2024},
eprint={2406.07302},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]

## Dataset Card Contact

[More Information Needed]

原始信息汇总

数据集概述

数据集描述

BertaQA 是一个包含 4,756 个多项选择题的 trivia 数据集，每个问题有一个正确答案和两个干扰项。问题分为本地和全球主题，本地问题需要关于巴斯克地区及其文化的特定知识，而全球问题需要更广泛的世界知识。问题还分为八个类别：巴斯克和文学、地理和历史、社会和传统、体育和休闲、文化和艺术、音乐和舞蹈、科学和技术、电影和节目。问题的难度分为简单、中等和困难。

数据集最初是用巴斯克语编制的，通过抓取现已不可用的公共来源。从原始的巴斯克语版本开始，我们还使用专业翻译服务创建了 BertaQA 的英语版本。翻译人员被指示为所有问题和答案使用一致的格式，并通过多轮修订我们的指南。对于命名实体，当可用时，使用维基百科作为参考。在翻译过程中，一些原始的巴斯克语问题被修正，要么是因为原始答案不正确，要么是因为它已经过时。此外，我们丢弃了一些需要巴斯克语或英语知识的问题，如果翻译，它们会失去本质。

最终的数据集在每个类别和子集的问题数量上是平衡的，每个类别大约有 300 个问题。每个难度的问数量也是平衡的：大多数类别在每个子集中大约有 110 个简单和中等难度的问题，以及 80 个困难的问题。问题和候选答案的平均长度分别约为 50 和 13 个字符。

• 语言： 巴斯克语 (eu), 英语 (en)
• 许可证： cc-by-4.0
• 大小类别： 1K<n<10K
• 任务类别： 问答

数据集结构

配置

• eu

  • 分割：test
  • 路径：eustrivia_zuzenduta.jsonl

• en

  • 分割：test
  • 路径：eustrivia_elhuyar_zuzenduta.jsonl

• en_mt_nllb

  • 分割：test
  • 路径：eustrivia_nllb_zuzenduta.jsonl

• en_mt_madlad

  • 分割：test
  • 路径：eustrivia_madlad_zuzenduta.jsonl

• en_mt_hitz

  • 分割：test
  • 路径：eustrivia_hitz_zuzenduta.jsonl

• en_mt_itzuli

  • 分割：test
  • 路径：eustrivia_itzuli_zuzenduta.jsonl

• en_mt_latxa-7b-v1.1

  • 分割：test
  • 路径：eustrivia_latxa-7b-v1.1_zuzenduta.jsonl

• en_mt_latxa-13b-v1.1

  • 分割：test
  • 路径：eustrivia_latxa-13b-v1.1_zuzenduta.jsonl

• en_mt_latxa-70b-v1.1

  • 分割：test
  • 路径：eustrivia_latxa-70b-v1.1_zuzenduta.jsonl

• en_mt_latxa-7b-v1

  • 分割：test
  • 路径：eustrivia_latxa-7b-v1_zuzenduta.jsonl

• en_mt_latxa-13b-v1

  • 分割：test
  • 路径：eustrivia_latxa-13b-v1_zuzenduta.jsonl

• en_mt_latxa-70b-v1

  • 分割：test
  • 路径：eustrivia_latxa-70b-v1_zuzenduta.jsonl

• en_mt_llama-2-7b

  • 分割：test
  • 路径：eustrivia_llama-2-7b_zuzenduta.jsonl

• en_mt_llama-2-13b

  • 分割：test
  • 路径：eustrivia_llama-2-13b_zuzenduta.jsonl

• en_mt_llama-2-70b

  • 分割：test
  • 路径：eustrivia_llama-2-70b_zuzenduta.jsonl

• en_mt_gemma-7b

  • 分割：test
  • 路径：eustrivia_gemma-7b_zuzenduta.jsonl

搜集汇总

数据集介绍

构建方式

BertaQA是一个精心构建的常识问答数据集，共包含4,756道多项选择题，每道题设有一个正确答案和两个干扰项。数据集最初通过爬取巴斯克语公开来源整理而成，并经过严格验证以确保原始数据不存在于现有模型的训练语料中。在此基础上，研究团队借助专业翻译服务创建了英文版本，翻译过程中对命名实体参照维基百科进行标准化，同时修正了原版中因时效性或准确性存在问题的题目，并剔除了依赖特定语言知识而无法保留原意的样本。最终的数据集在类别和子集间实现了数量均衡，每个类别约含300道题，难度分布亦保持平衡，题目与选项的平均长度分别约为50和13个字符。

使用方法

用户可通过HuggingFace Datasets库便捷地加载该数据集，支持选择巴斯克语原始版本、专业翻译的英文版本，以及由NLLB、MADLAD、Latxa、Llama-2、Gemma等多种机器翻译系统生成的英文译版。每个配置均以JSON Lines格式存储测试集数据，便于直接用于评估语言模型在常识问答任务上的表现。研究者可基于题目类别、难度或地域属性进行分层分析，尤其适合探究模型在本地化知识理解与跨语言迁移能力方面的差异。数据集还附有详细的论文与代码仓库链接，为复现实验和进一步研究提供了完整支持。

背景与挑战

背景概述

在自然语言处理领域，大型语言模型在知识密集型任务中展现出卓越能力，但其对特定地域文化的掌握程度仍是一个未充分探索的课题。为填补这一空白，HiTZ中心（隶属于巴斯克大学伊萨研究团队）于2024年创建了BertaQA数据集，旨在评估语言模型对巴斯克地区本土文化的理解深度。该数据集包含4,756道多项选择题，题目精心划分为本地与全球两大主题，并细分为八个文化类别及三个难度等级。研究团队通过爬取已不可公开访问的巴斯克语源数据，并借助专业翻译服务生成英文版本，确保了数据集的独特性与跨语言评估能力。BertaQA的发布为探究多语言模型在文化特异性知识上的表现提供了关键基准，其核心研究问题聚焦于模型是否能区分普遍世界知识与区域文化知识，对推动文化感知型NLP系统的发展具有深远影响。

当前挑战

BertaQA所解决的领域问题在于系统评估语言模型对地域文化知识的掌握程度，这超越了传统通用知识问答的范畴，要求模型具备精细的文化敏感性和跨语言理解能力。构建过程中面临的挑战尤为突出：首先，原始数据源自现已失效的巴斯克语公开资源，需通过严格去污染验证确保测试集未被模型预训练数据沾染，研究团队采用搜索引擎检索确认题目无法直接匹配。其次，题目翻译环节需兼顾文化保真度与格式一致性，专业译员需针对专有名词参照维基百科统一标准，并对部分过时或错误答案进行修正。此外，数据集在类别与难度维度上需保持平衡，最终实现每类约300题、难度分布均匀的精细调控，同时剔除依赖巴斯克语或英语语言知识而丧失原意的题目，这些步骤共同构成了数据集构建中的核心技术难点。

常用场景

经典使用场景

BertaQA数据集由HiTZ中心精心构建，包含4756道多项选择常识问答题目，每道题均配备一个正确答案与两个干扰项。其最经典的使用场景在于评估和比较语言模型对地方文化与全球知识的掌握程度。数据集巧妙地将问题划分为本地与全球两大主题，本地问题聚焦于巴斯克地区的独特文化与习俗，全球问题则考察通用世界知识。这种设计使得研究者能够系统性地剖析模型在文化特异性知识上的表现差异，为跨文化自然语言处理研究提供了极具价值的基准测试平台。

解决学术问题

该数据集精准回应了当前大语言模型研究中一个关键且尚未被充分探索的学术问题：模型在地方文化知识上的表征能力究竟如何？以往评估多侧重于通用知识或语言能力，忽视了文化多样性的影响。BertaQA通过引入文化维度，揭示了模型在处理地域性、小众化知识时可能存在的显著偏差与局限性。这一贡献不仅丰富了模型评估的维度，更推动学界重新审视训练数据的地域偏见问题，对构建更具包容性和文化敏感性的语言模型具有深远意义。

实际应用

在实际应用中，BertaQA为多语言问答系统的开发与优化提供了关键支撑。例如，面向巴斯克语社区的智能助手、文化教育平台或旅游信息服务，均可借助该数据集检验系统对当地用户真实需求的响应质量。此外，数据集所包含的机器翻译版本（如基于NLLB、MADLAD等模型生成的英文翻译）使得研究者能够系统性地评估翻译质量对下游问答任务的影响，从而指导跨语言信息检索和本地化服务的工程实践。

数据集最近研究

最新研究方向

BertaQA数据集的构建与评测聚焦于多语言大语言模型对地方性知识与全球性常识的掌握差异，尤其关注巴斯克文化等低资源语言区域的本地化理解能力。当前前沿研究方向涵盖跨语言知识迁移、机器翻译质量对问答性能的影响，以及大模型在文化特定语境下的偏见与鲁棒性评估。该数据集通过精细化的难度分级与八类主题划分，为探究模型在地方性知识（如巴斯克文学、传统习俗）与通用知识之间的表现鸿沟提供了独特视角。相关热点事件包括低资源语言问答基准的稀缺性挑战，以及大模型在多语种环境下的文化适应性研究。BertaQA的意义在于推动语言模型从通用能力向文化包容性、地域敏感性的纵深发展，为构建更公平、更具文化意识的人工智能系统奠定基础。