LumiOpen/lingsoft-pemt-evals
收藏资源简介：
--- language: - fi - en license: cc-by-4.0 task_categories: - translation - multiple-choice pretty_name: Lingsoft PEMT translation preference evals dataset_info: - config_name: FI_pairs features: - name: id dtype: string - name: task_id dtype: string - name: language_pair dtype: string - name: date dtype: string - name: segment dtype: string - name: segment_no dtype: string - name: source_text dtype: string - name: raw_mt dtype: string - name: target_pemt dtype: string - name: topics dtype: string splits: - name: test_shared num_bytes: 360233 num_examples: 527 - name: test_broad num_bytes: 2709525 num_examples: 4072 download_size: 1195504 dataset_size: 3069758 - config_name: FI_preference_5shot features: - name: id dtype: string - name: pair_id dtype: string - name: language dtype: string - name: split dtype: string - name: order dtype: string - name: shots dtype: int64 - name: prompt dtype: string - name: choices sequence: string - name: answer dtype: string - name: completion dtype: string - name: metadata struct: - name: option_a_source dtype: string - name: option_b_source dtype: string - name: segment dtype: string - name: segment_no dtype: string - name: task_id dtype: string splits: - name: test_shared num_bytes: 3015838 num_examples: 1054 - name: test_broad num_bytes: 23253850 num_examples: 8144 download_size: 2925824 dataset_size: 26269688 configs: - config_name: FI_pairs data_files: - split: test_shared path: FI_pairs/test_shared-* - split: test_broad path: FI_pairs/test_broad-* - config_name: FI_preference_5shot data_files: - split: test_shared path: FI_preference_5shot/test_shared-* - split: test_broad path: FI_preference_5shot/test_broad-* --- # Lingsoft PEMT translation preference evals Evaluation sets for translation quality and fluency, built from the [Lingsoft-EU-Summaries-PEMT](https://github.com/LumiOpen/Lingsoft-EU-Summaries-PEMT) corpus: professional post-edited machine translations of "Summaries of EU Legislation" (2016–2025). Each item pairs a raw machine translation (`Raw_MT`) with its professional human post-edit (`Target_PEMT`) for the same English source sentence. Only pairs where post-editing changed the text are included. Intended for use with the [LumiOpen lm-evaluation-harness](https://github.com/LumiOpen/lm-evaluation-harness) `lingsoft_pemt` tasks. ## Configs ### `FI_preference_5shot` Pre-rendered blind A/B preference prompts for Finnish (task `lingsoft_pemt_fi_mcf_*`): the prompt shows the English source and both translations labeled only "Finnish translation A/B", with 5 few-shot examples (sampled from the FI train split, balanced answer labels) baked into the prompt. Every pair appears twice with the A/B order swapped (`order` = `raw_first` / `target_first`), so random chance is exactly 50%. Score the single-token continuations `" A"` / `" B"` by loglikelihood; the correct answer is always the post-edited side. Generated by `create_fi_base_model_eval.py` in the source repo (5 shots, seed 42). ### `FI_pairs` The underlying changed rows (task `lingsoft_pemt_fi_cf_*`): `source_text`, `raw_mt`, `target_pemt` plus corpus metadata. `id` matches `pair_id` in `FI_preference_5shot`. Use for completion-form scoring — compare loglikelihoods of the two translations directly as continuations of the source. ## Splits - `test_shared`: text units from 6 segments shared across all 23 corpus languages (cross-lingually comparable if more languages are added). - `test_broad`: a broad sample across all segments. | config | test_shared | test_broad | |---|---|---| | FI_preference_5shot (records) | 1,054 | 8,144 | | FI_pairs (pairs) | 527 | 4,072 | ## Provenance and license Derived from professional translation and post-editing projects by Lingsoft under "Summaries of EU Legislation" (2016–2025). Original editorial content © European Union, re-used via the Publications Office of the European Union under CC-BY 4.0. Proprietary triple-stage alignment and raw MT layers provided by Lingsoft.
Lingsoft PEMT translation preference evals dataset is designed for evaluating translation quality and fluency. It is built from the Lingsoft-EU-Summaries-PEMT corpus, consisting of professional post-edited machine translation pairs where each item pairs a raw machine translation (Raw_MT) with its human post-edited version (Target_PEMT) for the same English source sentence, including only pairs where post-editing changed the text. The dataset is intended for use with the LumiOpen lm-evaluation-harness lingsoft_pemt tasks, supporting Finnish and English. It includes two configurations: FI_pairs provides the underlying translation pairs with metadata for completion-form scoring, and FI_preference_5shot offers pre-rendered blind A/B preference prompts with 5 few-shot examples for preference evaluation. Splits include test_shared (text units shared across 23 languages) and test_broad (a broad sample), with examples ranging from 527 to 8,144. Derived from professional translation and post-editing by Lingsoft under Summaries of EU Legislation (2016–2025), it is licensed under CC-BY 4.0.
