smartcat/STS_parallel_en_sr

---
license: mit
task_categories:
- text-classification
- sentence-similarity
- translation
language:
- en
- sr
tags:
- cross-lingual
- semantic-text-similarity
- sts-benchmark
- parallel-corpus
pretty_name: English-Serbian Semantic Text Similarity (STS) Benchmark
size_categories:
- n<1K
---

# Dataset Card for English-Serbian Semantic Text Similarity Benchmark

## Dataset Description

- **Repository:** [Provide a link to your dataset repository]

### Dataset Summary

This dataset is a parallel English-Serbian Semantic Text Similarity (STS) benchmark. It was created to evaluate multilingual English-Serbian language models, with a focus on SBERT (Sentence-BERT) knowledge distillation. The dataset consists of sentence pairs in English and Serbian, along with their semantic similarity scores.
The dataset uses the test split from the original STS benchmark. Only the second sentence of each pair was translated to Serbian. The translation was done automatically using the `gpt-3.5-turbo-0125 model`. The prompt used for the translation can be found in the repository.

### Supported Tasks and Leaderboards

- **Semantic Similarity:** The primary task for this dataset is to measure the semantic similarity between English-Serbian sentence pairs.
- **Cross-lingual Understanding:** This dataset can be used to evaluate models' performance in understanding semantic relationships across English and Serbian languages.
- **Machine Translation:** While not the primary focus, this dataset can also be used to assess English to Serbian translation quality.

### Languages

The dataset contains text in two languages:
- English (en)
- Serbian (sr)

## Dataset Structure

### Data Instances

Each instance in the dataset consists of:'
1. Datast split
2. Genre (the domain the sentence belongs to)
3. Dataset (original dataset a setence is obtained from)
4. Year (which itteration of the STS dataset a sentence belogs to)
5. Sid (a unique sentence id for each sentence pair)
7. Sentence 1 (an original English sentence from the STS benchmark datast)
8. Sentence 2 in English (an original English sentence from the STS benchmark datast)
9. Setnece 2 in Serbian (an automatically translated Sentence 2 from the STS benchmark dataset)
10. A similarity score (kept from the original STS benchmark dataset)

### Data Fields

- `en_sentence`: The original English sentence
- `sr_sentence`: The corresponding Serbian translation
- `similarity_score`: A float value representing the semantic similarity between the two sentences

### Data Splits

The dataset uses the test split from the original STS benchmark. Only the second sentence of each pair was translated to Serbian.

## Dataset Creation

### Curation Rationale

This dataset was created to address the need for multilingual semantic similarity benchmarks, specifically for the English-Serbian language pair. It aims to facilitate research in cross-lingual understanding and evaluation of multilingual language models.

### Source Data

The source data used for the translation is the test split of the STS benchmark dataset from: https://huggingface.co/datasets/mteb/stsbenchmark-sts

#### Initial Data Collection and Normalization

The source data comes from the test split of the original STS benchmark dataset.

## Considerations for Using the Data

### Discussion of Biases

As the Serbian translations were produced automatically, there may be biases or errors introduced by the machine translation process. Users should be aware of potential gender, cultural, or linguistic biases that may be present in the original English sentences or introduced during translation.

### Other Known Limitations

- The dataset is limited to the test split of the original STS benchmark, which may affect its size and diversity.
- The dataset may not cover all domains or types of semantic relationships.
- Automatic translation may not capture all nuances of the original English sentences. A random split of 100 translated sentences was validated. However, translation does not involve post-correction at this stage. Thus, the translated Serbian sentences might contain inconsistencies in relation to the rules and grammar of the standard Serbian language.

### Licensing Information

This dataset is published under the MIT license.

### Citation Information

If you use this dataset, please link the smartcat huggingface dataset repository.

## Loading the Dataset

To load this dataset using the Hugging Face `datasets` library, you can use the following code:

```python
from datasets import load_dataset

# Replace 'username/dataset_name' with the actual path to your dataset on Hugging Face
dataset = load_dataset("smartcat/STS_parallel_en_sr")

# Access the data
for example in dataset['train']:
print(f"Sentence 1: {example[sentence1]}")
print(f"Sentence 2 in English: {example['sentence2_eng']}")
print(f"Sentence 2 in Serbian: {example['sentence2_sr']}")
print(f"Similarity Score: {example['score']}")
print()

许可证：MIT
任务类别：
- 文本分类
- 句子相似度
- 翻译
语言：
- 英语（en）
- 塞尔维亚语（sr）
标签：
- 跨语言
- 语义文本相似度（Semantic Text Similarity，STS）
- STS基准
- 平行语料库
美观名称：英语-塞尔维亚语语义文本相似度（STS）基准数据集
大小类别：样本数少于1000

# 英语-塞尔维亚语语义文本相似度（STS）基准数据集卡片

## 数据集说明
- **仓库地址**：[请提供数据集仓库链接]

### 数据集摘要
本数据集为一套平行的英语-塞尔维亚语语义文本相似度（STS）基准数据集，旨在评估多语言英语-塞尔维亚语大语言模型，重点聚焦于SBERT（Sentence-BERT）知识蒸馏任务。数据集由英语与塞尔维亚语的句子对及其语义相似度得分构成。
本数据集沿用了原始STS基准数据集的测试划分，仅将每个句子对的第二句翻译为塞尔维亚语，翻译工作通过`gpt-3.5-turbo-0125`模型自动完成，翻译所用提示词可在仓库中获取。

### 支持任务与排行榜
- **语义相似度**：本数据集的核心任务为衡量英语-塞尔维亚语句子对之间的语义相似度。
- **跨语言理解**：该数据集可用于评估模型在理解英语与塞尔维亚语跨语言语义关联方面的性能。
- **机器翻译**：尽管并非核心目标，本数据集也可用于评估英语到塞尔维亚语的机器翻译质量。

### 语言分布
本数据集包含两种语言的文本：
- 英语（en）
- 塞尔维亚语（sr）

## 数据集结构
### 数据实例
数据集中的每个实例包含：
1. 数据集划分
2. 领域（句子所属的域）
3. 数据集来源（句子所属的原始数据集）
4. 年份（句子所属STS数据集的迭代版本）
5. Sid（每个句子对的唯一句子ID）
7. 句子1（源自STS基准数据集的原始英语句子）
8. 英语句子2（源自STS基准数据集的原始英语句子）
9. 塞尔维亚语句子2（源自STS基准数据集的句子2的自动翻译结果）
10. 相似度得分（沿用自原始STS基准数据集）

*注：原文存在序号跳变及拼写错误，翻译时保留原文格式与内容*

### 数据字段
- `en_sentence`：原始英语句子
- `sr_sentence`：对应的塞尔维亚语翻译
- `similarity_score`：代表两个句子间语义相似度的浮点数值

### 数据划分
本数据集沿用了原始STS基准数据集的测试划分，仅将每个句子对的第二句翻译为塞尔维亚语。

## 数据集构建
### 构建初衷
本数据集旨在满足跨语言语义相似度基准测试的需求，特别是针对英语-塞尔维亚语这一语言对，以期推动跨语言理解与多语言大语言模型评估相关研究。

### 源数据
用于翻译的源数据取自STS基准数据集的测试划分，来源链接：https://huggingface.co/datasets/mteb/stsbenchmark-sts

#### 初始数据收集与标准化
源数据取自原始STS基准数据集的测试划分。

## 数据集使用注意事项
### 偏差讨论
由于塞尔维亚语译文由机器自动生成，机器翻译流程可能引入偏差或错误。使用者需注意原始英语句子中可能存在的性别、文化或语言偏差，以及翻译过程中引入的潜在偏差。

### 其他已知局限性
- 本数据集仅限定于原始STS基准数据集的测试划分，这可能影响其规模与多样性。
- 本数据集可能未覆盖所有领域或类型的语义关联。
- 自动翻译可能无法完全还原原始英语句子的所有细微语义。本次已对100条随机抽取的翻译句子进行了验证，但目前尚未对翻译结果进行后期校正，因此塞尔维亚语译句可能存在与标准塞尔维亚语语法和规则不符的不一致之处。

### 许可证信息
本数据集采用MIT许可证发布。

### 引用信息
若使用本数据集，请关联smartcat的Hugging Face数据集仓库。

## 数据集加载方式
使用Hugging Face `datasets`库加载本数据集的代码如下：
python
from datasets import load_dataset

# 将 'username/dataset_name' 替换为你在Hugging Face上的实际数据集路径
dataset = load_dataset("smartcat/STS_parallel_en_sr")

# 访问数据
for example in dataset['train']:
print(f'Sentence 1: {example[sentence1]}')
print(f'Sentence 2 in English: {example["sentence2_eng"]}')
print(f'Sentence 2 in Serbian: {example["sentence2_sr"]}')
print(f'Similarity Score: {example["score"]}')
print()