ru-arena-hard

# ru-arena-hard
This is translated version of [arena-hard-auto](https://github.com/lmarena/arena-hard-auto?tab=readme-ov-file#style-control) dataset for evaluation LLMs. The translation of the original dataset was done manually. In addition, content of each task in dataset was reviewed, the correctness of the task statement and compliance with moral and ethical standards were assessed. Thus, this dataset allows you to evaluate the abilities of language models to support the Russian language.

### Overview of the Dataset
- Original dataset: [arena-hard-auto](https://github.com/lmarena/arena-hard-auto/blob/main/data/arena-hard-v0.1/question.jsonl)
- Number of tasks in original dataset: **500**
- Number of tasks: **474**
- Format: **JSON LINES**

### Usage
To use this dataset for model estimation, follow these steps:

1. Download [`question.jsonl`](https://huggingface.co/datasets/t-tech/ru-arena-hard/blob/main/data/question.jsonl) and [`gpt-4-0613.jsonl`](https://huggingface.co/datasets/t-tech/ru-arena-hard/blob/main/gpt_baseline_responses/gpt-4-0613.jsonl) files.
2. Then use file `question.jsonl` instead of `arena-hard-auto/data/arena-hard-v0.1/question.jsonl` and file `gpt-4-0613.jsonl` instead of `arena-hard-auto/data/arena-hard-v0.1/model_answer/gpt-4-0613.jsonl` with [original codebase](https://github.com/lmarena/arena-hard-auto/tree/main).

### Sample example

```json
{
"question_id": "328c149ed45a41c0b9d6f14659e63599", "category": "arena-hard-v0.1",
"cluster": "ABC Sequence Puzzles & Groups",
"turns": [{"content": "Напишите мелодию в стиле фолк с помощью ABC-нотации"}]
}
```

# ru-arena-hard
本数据集为用于评估大语言模型（Large Language Model，LLM）的[arena-hard-auto](https://github.com/lmarena/arena-hard-auto?tab=readme-ov-file#style-control)数据集的俄语翻译版本。原始数据集的翻译工作由人工完成。此外，本数据集内的每一项任务均经过内容审核，对任务描述的准确性以及是否符合道德伦理标准进行了评估。借此，本数据集可用于评估语言模型对俄语的支持能力。

### 数据集概览
- 原始数据集：[arena-hard-auto](https://github.com/lmarena/arena-hard-auto/blob/main/data/arena-hard-v0.1/question.jsonl)
- 原始数据集任务总量：**500**
- 本数据集任务总量：**474**
- 数据格式：**JSON LINES**

### 使用方法
若使用本数据集进行模型评估，请遵循以下步骤：
1. 下载[`question.jsonl`](https://huggingface.co/datasets/t-tech/ru-arena-hard/blob/main/data/question.jsonl)与[`gpt-4-0613.jsonl`](https://huggingface.co/datasets/t-tech/ru-arena-hard/blob/main/gpt_baseline_responses/gpt-4-0613.jsonl)两个文件。
2. 在使用[原始代码库](https://github.com/lmarena/arena-hard-auto/tree/main)时，将`arena-hard-auto/data/arena-hard-v0.1/question.jsonl`替换为`question.jsonl`，并将`arena-hard-auto/data/arena-hard-v0.1/model_answer/gpt-4-0613.jsonl`替换为`gpt-4-0613.jsonl`。

### 数据样例
json
{
"question_id": "328c149ed45a41c0b9d6f14659e63599", "category": "arena-hard-v0.1",
"cluster": "ABC Sequence Puzzles & Groups",
"turns": [{"content": "Напишите мелодию в стиле фолк с помощью ABC-нотации"}]
}