bouquet|机器翻译数据集|质量评估数据集

# BOUQuET 💐: Benchmark and Open initiative for Universal Quality Evaluation in Translation

BOUQuET is a multi-way parallel, multi-centric and multi-register/domain dataset and benchmark for machine translation quality.
The underlying texts have been handcrafted by linguists in 8 diverse languages and translated to English, and the dataset is intended to be extensible to virtually any other written language.
BOUQuET has been described in the paper by [Omnilingual Team, 2025](https://arxiv.org/abs/2502.04314) and can be extended by contributing new translations in https://bouquet.metademolab.com.

## Dataset Details

### Dataset Description

- **Language(s) (NLP):** multilingual
- **Home:** https://bouquet.metademolab.com/
- **Paper:** https://arxiv.org/abs/2502.04314

## Uses

The dataset is intended for evaluation of machine translation quality.
By purpose, it is similar to [FLORES+](https://huggingface.co/datasets/openlanguagedata/flores_plus) or [WMT24++](https://huggingface.co/datasets/google/wmt24pp).
Unlike these datasets, BOUQuET focuses more on linguistic diversity.

It is not intended as a training dataset, but the `dev` subset may be used for validation during model development.

## Dataset Structure

### Composition
BOUQuET consists of short paragraphs, fully parallel in all languages at the sentence level.
The dataset is distributed both at the sentence level and at the paragraph level.
By default, only sentence-level data is loaded; the `paragraph_level` config may be used to load the paragraph-level data.

For convenience, every language is paired with English, but actually the dataset is fully multi-way parallel, so any language could be paired with any other.

The public portion of the dataset contains two splits:
- `dev`: 504 unique sentences, 120 paragraphs
- `test`: 864 unique sentences, 200 paragraphs
An additional split made up of 632 unique sentences and 144 paragraphs is being held out for quality assurance purposes and is not distributed here.

### Columns

The dataset contains the following fields:
```
- level # str, "sentence_level" or "paragraph_level"
- split # str, "dev" or "test"
- uniq_id # str, identifier of the dataset item (e.g. `P464-S1` for sentence-level, `P464` for paragraph-level data)
- src_lang # str, NLLB-compatible non-English language code (such as `hin_Deva`)
- tgt_lang # str, "eng_Latn"
- src_text # str, non-English text
- tgt_text # str, English text
- orig_text # str, the original text (sentence or paragraph), which sometimes corresponds to src_text
- par_comment # str, comment to the whole paragraph
- sent_comment # str, comment to the sentence
- has_hashtag # bool, whether a hash tag is present in the text
- has_emoji # bool, whether an emoji is present in the text
- has_12p # bool, whether the sentence has first- or second-person pronouns
- has_speaker_tag # bool, whether the sentence starts with a speaker tag
- newline_next # bool, whether the sentence should be followed by a newline in the paragraph
- par_id # str, paragraph id (e.g. `P464`)
- domain # str, one of the 8 domains (see the paper)
- register # str, three-letter identifier of the register (see the paper)
- tags # str, comma-separated linguistic tags of a sentence (see the paper)
```

### Languages

Currently, BOUQuET covers 9 language varieties:

| ISO 639-3 | ISO 15924 | Language | Family | Subgroup1 |
|--|--|--|--|--|
arz | Arab | Egyptian Arabic | Afro-Asiatic | West Semitic |
cmn | Hans | Mandarin Chinese | Sino-Tibetan | Sinitic |
deu | Latn | German | Indo-European | West Germanic |
eng | Latn | English | Indo-European | West Germanic |
fra | Latn | French | Indo-European | Italic |
hin | Deva | Hindi | Indo-European | Indo-Aryan |
ind | Latn | Indonesian | Austronesian | Malayic |
rus | Cyrl | Russian | Indo-European | Balto-Slavic |
spa | Latn | Spanish | Indo-European | Italic |

Each language variety is characterized by an ISO 639-3 code for its language, and an ISO 15924 code for the writing system.

To contribute translations for new languages, please use our crowdsourcing tool: https://bouquet.metademolab.com.

## Usage examples

The code below loads a pre-configured subset, French sentences paired with English, and selects the first instance
```Python
import datasets
data = datasets.load_dataset("facebook/bouquet", "fra_Latn", split="dev")
# to demonstrate an example, we select a single data instance
data[0]
# {'uniq_id': 'P037-S1',
# 'src_lang': 'fra_Latn',
# 'src_text': 'Tu as des mains en or, la nourriture est délicieuse.',
# 'tgt_lang': 'eng_Latn',
# 'domain': 'comments',
# 'tgt_text': 'Bless your hands, the food was very delicious. ',
# 'par_comment': 'possessive pronoun "your" is 2nd person feminine',
# 'tags': 'second person, single tense (past)',
# 'register': 'mra',
# 'orig_text': 'تسلم ايديكي الاكل كان جميل جدًا',
# 'has_speaker_tag': False,
# 'has_hashtag': False,
# 'has_emoji': False,
# 'has_12p': True,
# 'newline_next': True,
# 'level': 'sentence_level',
# 'split': 'dev',
# 'par_id': 'P037',
# 'sent_comment': None}
```

Another example loads paragraph-level data paired with English, and then pairs Spanish sentences with their Russian translations:

```Python
import datasets
data = datasets.load_dataset("facebook/bouquet", "paragraph_level", split="dev").to_pandas()

spa2rus = pd.merge(
data.loc[data["src_lang"].eq("spa_Latn")].drop(["tgt_lang", "tgt_text"], axis=1),
data.loc[data["src_lang"].eq("rus_Cyrl"), ["src_lang", "src_text", "uniq_id"]].rename({"src_lang": "tgt_lang", "src_text": "tgt_text"}, axis=1),
on="uniq_id",
)
```

## Dataset Creation

### Curation Rationale

The dataset has been created manually from scratch, by composing the source sentences that cover a variety of domains and registers in 8 diverse non-English languages: Egyptian Arabic (alternating with Modern Standard Arabic when appropriate), French, German, Hindi, Indonesian, Mandarin Chinese, Russian, and Spanish.

For each of the source languages, the sentences have been created in the following 8 domains:
1. How-to, written tutorials or instructions
2. Conversations (dialogues)
3. Narration (creative writing that doesn’t include dialogues)
4. Social media posts
5. Social media comments (reactive)
6. Other web content
7. Reflective piece
8. Miscellaneous (address to a nation, disaster response, etc.)

Apart from the domains, a variety of registers (contextual styles) were used. Each sentence is annotated with the register characterized by three features: connectedness, preparedness, and social differential.

The linguists who were creating the dataset were instructed to maintain the diversity of sentence lengths, word orders, sentence structures, and other linguistic characteristics.

Subsequently, the source sentences were translated from the 8 source languages into English. We plan to extend the dataset "in width", by translating it into even more languages.

See the [paper](https://arxiv.org/abs/2502.04314) for more details.

## Contribution

To contribute to the dataset (adding translations for a new language, or verifying some of the existing translations), please use the web annotation tool at https://bouquet.metademolab.com.

## Citation

If you are referring to this dataset, please cite the [BOUQuET paper](https://arxiv.org/abs/2502.04314).

```bibtex
@article{andrews2025bouquet,
title={BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation},
author={{The Omnilingual MT Team} and Andrews, Pierre and Artetxe, Mikel and Meglioli, Mariano Coria and Costa-juss{\`a}, Marta R and Chuang, Joe and Dupenthaler, Mark and Dale, David and Ekberg, Nate and Gao, Cynthia and Maillard, Jean and Licht, Daniel and Mourachko, Alex and Ropers, Christophe and Saleem, Safiyyah and Sánchez, Eduardo, and Tsiamas, Ioannis, and Turkatenko, Arina and Ventayol, Albert and Yates, Shireen},
journal={arXiv preprint arXiv:2502.04314},
year={2025}
}
```

## Glossary

- **Domain.** By the term *domain*, we mean different spaces in which language is produced in speech, sign, or writing (e.g., books, social media, news, Wikipedia, organization websites, official documents, direct messaging, texting). In this paper, we focus solely on the written modality.
- **Register.** We understand the term *register* as a functional variety of language that includes socio-semiotic properties, as expressed in [Halliday and Matthiessen (2004)], or more simply as a "contextual style", as presented in [Labov (1991), pp.79–99]. In that regard, a register is a specific variety of language used to best fit a specific communicative purpose in a specific situation.