argilla/mistral-vs-llama-rating

---
size_categories: n<1K
tags:
- rlfh
- argilla
- human-feedback
---

# Dataset Card for mistral-vs-llama-rating

This dataset has been created with [Argilla](https://docs.argilla.io).

As shown in the sections below, this dataset can be loaded into Argilla as explained in [Load with Argilla](#load-with-argilla), or used directly with the `datasets` library in [Load with `datasets`](#load-with-datasets).

## Dataset Description

- **Homepage:** https://argilla.io
- **Repository:** https://github.com/argilla-io/argilla
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

This dataset contains:

* A dataset configuration file conforming to the Argilla dataset format named `argilla.yaml`. This configuration file will be used to configure the dataset when using the `FeedbackDataset.from_huggingface` method in Argilla.

* Dataset records in a format compatible with HuggingFace `datasets`. These records will be loaded automatically when using `FeedbackDataset.from_huggingface` and can be loaded independently using the `datasets` library via `load_dataset`.

* The [annotation guidelines](#annotation-guidelines) that have been used for building and curating the dataset, if they've been defined in Argilla.

### Load with Argilla

To load with Argilla, you'll just need to install Argilla as `pip install argilla --upgrade` and then use the following code:

```python
import argilla as rg

ds = rg.FeedbackDataset.from_huggingface("argilla/mistral-vs-llama-rating")
```

### Load with `datasets`

To load this dataset with `datasets`, you'll just need to install `datasets` as `pip install datasets --upgrade` and then use the following code:

```python
from datasets import load_dataset

ds = load_dataset("argilla/mistral-vs-llama-rating")
```

### Supported Tasks and Leaderboards

This dataset can contain [multiple fields, questions and responses](https://docs.argilla.io/en/latest/guides/llms/conceptual_guides/data_model.html) so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the [Dataset Structure section](#dataset-structure).

There are no leaderboards associated with this dataset.

### Languages

[More Information Needed]

## Dataset Structure

### Data in Argilla

The dataset is created in Argilla with: **fields**, **questions**, **suggestions**, and **guidelines**.

The **fields** are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.

| Field Name | Title | Type | Required | Markdown |
| ---------- | ----- | ---- | -------- | -------- |
| input | Input | TextField | True | False |
| response_1 | Response_1 | TextField | True | True |
| response_2 | Response_2 | TextField | True | True |

The **questions** are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.

| Question Name | Title | Type | Required | Description | Values/Labels |
| ------------- | ----- | ---- | -------- | ----------- | ------------- |
| rating_1 | Rate the response_1? | RatingQuestion | True | N/A | [1, 2, 3, 4, 5] |
| rationale_1 | Rationale behind response_1's ranking? | TextQuestion | True | N/A | N/A |
| rating_2 | Rate the response_2? | RatingQuestion | True | N/A | [1, 2, 3, 4, 5] |
| rationale_2 | Rationale behind response_2's ranking? | TextQuestion | True | N/A | N/A |

**✨ NEW** Additionally, we also have **suggestions**, which are linked to the existing questions, and so on, named appending "-suggestion" and "-suggestion-metadata" to those, containing the value/s of the suggestion and its metadata, respectively. So on, the possible values are the same as in the table above.

Finally, the **guidelines** are just a plain string that can be used to provide instructions to the annotators. Find those in the [annotation guidelines](#annotation-guidelines) section.

### Data Instances

An example of a dataset instance in Argilla looks as follows:

```json
{
"fields": {
"input": "How do I locate my card?",
"response_1": " Thank you for reaching out! I\u0027m here to help you with your query. However, I must point out that the term \"locate\" can have different meanings depending on the context. Could you please provide more details or clarify what you mean by \"locate\"? Are you trying to find a specific card, such as a credit card or debit card? Or perhaps you\u0027re looking for something else entirely? Once I have a better understanding of your question, I\u0027ll be happy to assist you in a responsible and accurate manner.",
"response_2": " Of course! I\u0027d be happy to help you locate your missing card. To start, could you please tell me more about the card you\u0027re trying to find? Is it a credit card, debit card, or another type of card? Additionally, can you provide more details about the circumstances surrounding its disappearance? For example, did you lose it at home, in a store, or somewhere else? Any additional information you can provide will help me give you more tailored advice on how to locate your card."
},
"metadata": {},
"responses": [
{
"status": "submitted",
"user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd",
"values": {
"rating_1": {
"value": 3
},
"rating_2": {
"value": 4
},
"rationale_1": {
"value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence it\u0027s not highly informative."
},
"rationale_2": {
"value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful."
}
}
}
],
"suggestions": [
{
"agent": "gpt-4",
"question_id": "0a3cf946-2bd3-4d57-b42c-caddf8cc4979",
"question_name": "rating_1",
"value": 3
},
{
"agent": "gpt-4",
"question_id": "d7d8347b-554a-4eaf-ad2b-64898f3e0841",
"question_name": "rationale_1",
"value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence it\u0027s not highly informative."
},
{
"agent": "gpt-4",
"question_id": "99672a09-0d1c-45d7-ac2c-fe692603c5b4",
"question_name": "rating_2",
"value": 4
},
{
"agent": "gpt-4",
"question_id": "8276bff4-1106-4fef-abcb-ebc70beb791e",
"question_name": "rationale_2",
"value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful."
}
]
}
```

While the same record in HuggingFace `datasets` looks as follows:

```json
{
"external_id": null,
"input": "How do I locate my card?",
"metadata": "{}",
"rating_1": [
{
"status": "submitted",
"user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd",
"value": 3
}
],
"rating_1-suggestion": 3,
"rating_1-suggestion-metadata": {
"agent": "gpt-4",
"score": null,
"type": null
},
"rating_2": [
{
"status": "submitted",
"user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd",
"value": 4
}
],
"rating_2-suggestion": 4,
"rating_2-suggestion-metadata": {
"agent": "gpt-4",
"score": null,
"type": null
},
"rationale_1": [
{
"status": "submitted",
"user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd",
"value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence it\u0027s not highly informative."
}
],
"rationale_1-suggestion": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence it\u0027s not highly informative.",
"rationale_1-suggestion-metadata": {
"agent": "gpt-4",
"score": null,
"type": null
},
"rationale_2": [
{
"status": "submitted",
"user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd",
"value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful."
}
],
"rationale_2-suggestion": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful.",
"rationale_2-suggestion-metadata": {
"agent": "gpt-4",
"score": null,
"type": null
},
"response_1": " Thank you for reaching out! I\u0027m here to help you with your query. However, I must point out that the term \"locate\" can have different meanings depending on the context. Could you please provide more details or clarify what you mean by \"locate\"? Are you trying to find a specific card, such as a credit card or debit card? Or perhaps you\u0027re looking for something else entirely? Once I have a better understanding of your question, I\u0027ll be happy to assist you in a responsible and accurate manner.",
"response_2": " Of course! I\u0027d be happy to help you locate your missing card. To start, could you please tell me more about the card you\u0027re trying to find? Is it a credit card, debit card, or another type of card? Additionally, can you provide more details about the circumstances surrounding its disappearance? For example, did you lose it at home, in a store, or somewhere else? Any additional information you can provide will help me give you more tailored advice on how to locate your card."
}
```

### Data Fields

Among the dataset fields, we differentiate between the following:

* **Fields:** These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.

* **input** is of type `TextField`.
* **response_1** is of type `TextField`.
* **response_2** is of type `TextField`.

* **Questions:** These are the questions that will be asked to the annotators. They can be of different types, such as `RatingQuestion`, `TextQuestion`, `LabelQuestion`, `MultiLabelQuestion`, and `RankingQuestion`.

* **rating_1** is of type `RatingQuestion` with the following allowed values [1, 2, 3, 4, 5].
* **rationale_1** is of type `TextQuestion`.
* **rating_2** is of type `RatingQuestion` with the following allowed values [1, 2, 3, 4, 5].
* **rationale_2** is of type `TextQuestion`.

* **✨ NEW** **Suggestions:** As of Argilla 1.13.0, the suggestions have been included to provide the annotators with suggestions to ease or assist during the annotation process. Suggestions are linked to the existing questions, are always optional, and contain not just the suggestion itself, but also the metadata linked to it, if applicable.

* (optional) **rating_1-suggestion** is of type `rating` with the following allowed values [1, 2, 3, 4, 5].
* (optional) **rationale_1-suggestion** is of type `text`.
* (optional) **rating_2-suggestion** is of type `rating` with the following allowed values [1, 2, 3, 4, 5].
* (optional) **rationale_2-suggestion** is of type `text`.

Additionally, we also have one more field which is optional and is the following:

* **external_id:** This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file.

### Data Splits

The dataset contains a single split, which is `train`.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation guidelines

[More Information Needed]

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]

原始信息汇总

数据集卡片 for mistral-vs-llama-rating

数据集描述

数据集概述

该数据集包含：

• 符合 Argilla 数据集格式的配置文件 argilla.yaml，用于在使用 FeedbackDataset.from_huggingface 方法时配置数据集。
• 兼容 HuggingFace datasets 格式的数据记录，这些记录在使用 FeedbackDataset.from_huggingface 时会自动加载，也可以通过 datasets 库独立加载。
• 用于构建和整理数据集的标注指南（如果已在 Argilla 中定义）。

加载方式

使用 Argilla 加载

安装 Argilla 后，使用以下代码加载数据集：

python import argilla as rg

ds = rg.FeedbackDataset.from_huggingface("argilla/mistral-vs-llama-rating")

使用 datasets 加载

安装 datasets 后，使用以下代码加载数据集：

python from datasets import load_dataset

ds = load_dataset("argilla/mistral-vs-llama-rating")

支持的任务和排行榜

该数据集可以包含多个字段、问题和响应，因此可用于不同的 NLP 任务，具体取决于配置。数据集结构在数据集结构部分中描述。

该数据集没有关联的排行榜。

语言

[更多信息需要]

数据集结构

数据在 Argilla 中

数据集在 Argilla 中创建，包含：字段、问题、建议和指南。

字段是数据记录本身，目前仅支持文本字段。这些字段将用于提供对问题的响应。

字段名称	标题	类型	必需	Markdown
input	输入	TextField	True	False
response_1	响应_1	TextField	True	True
response_2	响应_2	TextField	True	True

问题是向标注者提出的问题。它们可以是不同类型，如评分、文本、单选或多选。

问题名称	标题	类型	必需	描述	值/标签
rating_1	评价响应_1？	RatingQuestion	True	N/A	[1, 2, 3, 4, 5]
rationale_1	响应_1排名的理由？	TextQuestion	True	N/A	N/A
rating_2	评价响应_2？	RatingQuestion	True	N/A	[1, 2, 3, 4, 5]
rationale_2	响应_2排名的理由？	TextQuestion	True	N/A	N/A

✨ NEW 此外，还有建议，这些建议与现有问题相关联，并命名为在问题名称后附加 "-suggestion" 和 "-suggestion-metadata"，包含建议的值及其元数据。可能的值与上表相同。

最后，指南是一个纯字符串，可用于向标注者提供指令。请参阅标注指南部分。

数据实例

在 Argilla 中的数据集实例示例如下：

json { "fields": { "input": "How do I locate my card?", "response_1": " Thank you for reaching out! Iu0027m here to help you with your query. However, I must point out that the term "locate" can have different meanings depending on the context. Could you please provide more details or clarify what you mean by "locate"? Are you trying to find a specific card, such as a credit card or debit card? Or perhaps youu0027re looking for something else entirely? Once I have a better understanding of your question, Iu0027ll be happy to assist you in a responsible and accurate manner.", "response_2": " Of course! Iu0027d be happy to help you locate your missing card. To start, could you please tell me more about the card youu0027re trying to find? Is it a credit card, debit card, or another type of card? Additionally, can you provide more details about the circumstances surrounding its disappearance? For example, did you lose it at home, in a store, or somewhere else? Any additional information you can provide will help me give you more tailored advice on how to locate your card." }, "metadata": {}, "responses": [ { "status": "submitted", "user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd", "values": { "rating_1": { "value": 3 }, "rating_2": { "value": 4 }, "rationale_1": { "value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence itu0027s not highly informative." }, "rationale_2": { "value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful." } } } ], "suggestions": [ { "agent": "gpt-4", "question_id": "0a3cf946-2bd3-4d57-b42c-caddf8cc4979", "question_name": "rating_1", "value": 3 }, { "agent": "gpt-4", "question_id": "d7d8347b-554a-4eaf-ad2b-64898f3e0841", "question_name": "rationale_1", "value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence itu0027s not highly informative." }, { "agent": "gpt-4", "question_id": "99672a09-0d1c-45d7-ac2c-fe692603c5b4", "question_name": "rating_2", "value": 4 }, { "agent": "gpt-4", "question_id": "8276bff4-1106-4fef-abcb-ebc70beb791e", "question_name": "rationale_2", "value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful." } ] }

在 HuggingFace datasets 中的相同记录示例如下：

json { "external_id": null, "input": "How do I locate my card?", "metadata": "{}", "rating_1": [ { "status": "submitted", "user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd", "value": 3 } ], "rating_1-suggestion": 3, "rating_1-suggestion-metadata": { "agent": "gpt-4", "score": null, "type": null }, "rating_2": [ { "status": "submitted", "user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd", "value": 4 } ], "rating_2-suggestion": 4, "rating_2-suggestion-metadata": { "agent": "gpt-4", "score": null, "type": null }, "rationale_1": [ { "status": "submitted", "user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd", "value": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence itu0027s not highly informative." } ], "rationale_1-suggestion": "The text is correct and provides a helpful response to the query. However, it lacks specific instructions or steps to help the user locate their card, hence itu0027s not highly informative.", "rationale_1-suggestion-metadata": { "agent": "gpt-4", "score": null, "type": null }, "rationale_2": [ { "status": "submitted", "user_id": "39bdc103-0e37-4515-a353-abe3b68bdebd", "value": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful." } ], "rationale_2-suggestion": "The text is accurate and provides a more detailed response to the query. It asks for more specific information to provide a tailored solution, making it highly informative. However, it still lacks the profound insights or comprehensive information that would make it outstandingly helpful.", "rationale_2-suggestion-metadata": { "agent": "gpt-4", "score": null, "type": null }, "response_1": " Thank you for reaching out! Iu0027m here to help you with your query. However, I must point out that the term "locate" can have different meanings depending on the context. Could you please provide more details or clarify what you mean by "locate"? Are you trying to find a specific card, such as a credit card or debit card? Or perhaps youu0027re looking for something else entirely? Once I have a better understanding of your question, Iu0027ll be happy to assist you in a responsible and accurate manner.", "response_2": " Of course! Iu0027d be happy to help you locate your missing card. To start, could you please tell me more about the card youu0027re trying to find? Is it a credit card, debit card, or another type of card? Additionally, can you provide more details about the circumstances surrounding its disappearance? For example, did you lose it at home, in a store, or somewhere else? Any additional information you can provide will help me give you more tailored advice on how to locate your card." }

数据字段

数据集字段包括：

• 字段：这些是数据记录本身，目前仅支持文本字段。这些字段将用于提供对问题的响应。

  • input 是 TextField 类型。
  • response_1 是 TextField 类型。
  • response_2 是 TextField 类型。

• 问题：这些是向标注者提出的问题。它们可以是不同类型，如 RatingQuestion、TextQuestion、LabelQuestion、MultiLabelQuestion 和 RankingQuestion。

  • rating_1 是 RatingQuestion 类型，允许值为 [1, 2, 3, 4, 5]。
  • rationale_1 是 TextQuestion 类型。
  • rating_2 是 RatingQuestion 类型，允许值为 [1, 2, 3, 4, 5]。
  • rationale_2 是 TextQuestion 类型。

• ✨ NEW 建议：自 Argilla 1.13.0 起，建议已包含在内，以向标注者提供建议，以简化或协助标注过程。建议与现有问题相关联，总是可选的，并且不仅包含建议本身，还包含其相关元数据（如果适用）。

  • (可选) rating_1-suggestion 是 rating 类型，允许值为 [1, 2, 3, 4, 5]。
  • (可选) rationale_1-suggestion 是 text 类型。
  • (可选) rating_2-suggestion 是 rating 类型，允许值为 [1, 2, 3, 4, 5]。
  • (可选) rationale_2-suggestion 是 text 类型。

此外，还有一个可选字段：

• external_id：这是一个可选字段，可用于为数据记录提供外部 ID。如果您想将数据记录链接到外部资源（如数据库或文件），这可能很有用。

数据分割

数据集包含一个分割，即 train。