argilla/kto-mix-15k

---
language:
- en
license: mit
size_categories:
- 1K<n<10K
configs:
- config_name: default
data_files:
- split: train
path: data/train-*
- split: test
path: data/test-*
dataset_info:
features:
- name: prompt
dtype: string
- name: completion
list:
- name: content
dtype: string
- name: role
dtype: string
- name: label
dtype: bool
- name: rating
dtype: float64
- name: dataset
dtype: string
splits:
- name: train
num_bytes: 53414380
num_examples: 14549
download_size: 20250148
dataset_size: 53414380
tags:
- distilabel
- synthetic
- kto
---

# Argilla KTO Mix 15K Dataset

> A KTO signal transformed version of the highly loved [Argilla DPO Mix](https://huggingface.co/datasets/argilla/dpo-mix-7k), which is small cocktail combining DPO datasets built by Argilla with [distilabel](https://github.com/argilla-io/distilabel). The goal of this dataset is having a small, high-quality KTO dataset by filtering only highly rated chosen responses.

<div>
<img src="https://cdn-uploads.huggingface.co/production/uploads/60420dccc15e823a685f2b03/Csd2-zPji7iwIxyz6UFe1.webp">
</div>

<p align="center">
<a href="https://github.com/argilla-io/distilabel">
<img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
</a>
</p>

## Why KTO?

The [KTO paper](https://arxiv.org/abs/2402.01306) states:

- KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of n DPO pairs and breaking it up into 2n examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal.
- KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
- When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.

## Reproduce KTO Transformation

[Original DPO dataset: DPO mix 7k](https://huggingface.co/datasets/argilla/dpo-mix-7k)

<a target="_blank" href="https://colab.research.google.com/drive/10bMnI3vvG4hEKblUhtLZKu01YSnPhmaF?usp=sharing">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

原始信息汇总

数据集概述

基本信息

• 名称: Argilla KTO Mix 15K Dataset
• 语言: 英语
• 许可证: MIT
• 大小分类: 1K<n<10K

配置

• 默认配置:
  • 训练数据路径: data/train-*
  • 测试数据路径: data/test-*

数据集信息

• 特征:
  • prompt: 字符串类型
  • completion:
    • content: 字符串类型
    • role: 字符串类型
  • label: 布尔类型
  • rating: 浮点64类型
  • dataset: 字符串类型
• 分割:
  • 训练:
    • 字节数: 53414380
    • 示例数: 14549
• 下载大小: 20250148
• 数据集大小: 53414380

标签

• distilabel
• synthetic
• kto