kardosdrur/europarl-scandinavian

---
license: mit
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: test
path: data/test-*
dataset_info:
features:
- name: da
dtype: string
- name: en
dtype: string
- name: sv
dtype: string
splits:
- name: train
num_bytes: 620348322.4
num_examples: 1304296
- name: test
num_bytes: 155087080.6
num_examples: 326074
download_size: 488376564
dataset_size: 775435403.0
---

# Europarl Scandinavian Languages

The data originates from the Europarl parallel corpus, where English transcriptions of parliamentary discussions were aligned
with a number of other languages algorithmically.

In order to align Danish and Swedish corpora in the dataset, English entries were hashed with 128bit Murmurhash3,
and the Danish and Swedish transcriptions were joined on the obtained hash values.
Entries that had more than one pair in the other dataset were removed, this ensures that no false positives due to hash collisions
got into the dataset.
Source code is available in the repository.

The dataset was created for aiding the training of sentence transformer models in the Danish Foundation Models project.

原始信息汇总

Europarl Scandinavian Languages 数据集概述

数据集信息

许可证

• MIT 许可证

配置

• 默认配置 (default)
  • 数据文件路径：
    • 训练集 (train): data/train-*
    • 测试集 (test): data/test-*

特征

• 特征名称及数据类型：
  • da: 字符串 (string)
  • en: 字符串 (string)
  • sv: 字符串 (string)

数据分割

• 训练集 (train)
  • 字节数: 620,348,322.4
  • 样本数: 1,304,296
• 测试集 (test)
  • 字节数: 155,087,080.6
  • 样本数: 326,074

数据集大小

• 下载大小: 488,376,564 字节
• 数据集大小: 775,435,403.0 字节

数据集来源

• 数据源自 Europarl 平行语料库，通过算法将英语议会讨论的转录与多种其他语言对齐。
• 为了对齐数据集中的丹麦语和瑞典语语料，英语条目通过 128bit Murmurhash3 进行哈希处理，丹麦语和瑞典语转录根据获得的哈希值进行连接。
• 删除在另一个数据集中有多个配对的条目，以确保由于哈希冲突导致的假阳性不会进入数据集。
• 数据集用于支持丹麦基础模型项目中句子转换模型的训练。