ChatGPT vs. DeepL vs. Google Translate: a human evaluation of multiword expressions' machine translation quality

This dataset supports the study “ChatGPT vs. DeepL vs. Google Translate: A human evaluation of multiword expression’s machine translation quality” and provides an open benchmark for analysing how MT systems handle Spanish multiword expressions (MWEs), including both continuous and discontinuous forms.

What the dataset contains

The dataset includes 600 Spanish-to-English translation instances featuring 6 Spanish idiomatic MWEs, with:

100 examples per MWE

50 continuous + 50 discontinuous examples per MWE

Real-world contextualised usage (each entry contains the MWE in context)

Each example is translated by three widely-used systems:

DeepL (Classic model / NMT mode)

Google Translate

GPT-4o (ChatGPT)

Human evaluation labels

Each MT output is manually evaluated using a binary scoring scheme focused specifically on whether the MWE is translated correctly and idiomatically:

1 = correct translation of the MWE

0 = incorrect translation of the MWE (e.g., literal or wrong meaning)

For discontinuous MWEs, translations are only marked as correct if they:

capture the idiomatic meaning, and

preserve the intervening material appropriately (i.e., no omission of key inserted elements).

File format and structure

The dataset is provided as a single Excel file:

MonTi paper - Dataset.xlsx

It contains 7 sheets:

Analysis (summary scores per system and MWE, split by continuous/discontinuous)

(Ser) cuatro gatos

Haber gato encerrado

Llevarse el gato al agua

Pagar el pato

Pagar/Costar cuatro perras

Dormir la mona

Each MWE sheet includes the following columns:

Source: Spanish source segment (with context)

DEEPL: DeepL translation

Result: DeepL binary evaluation (0/1)

GT: Google Translate translation

Result 2: Google Translate binary evaluation (0/1)

CHATGPT: GPT-4o translation

Result 3: GPT-4o binary evaluation (0/1)

Related publication

This dataset is released as the research data accompanying the paper mentioned above.