---
license: cc-by-nc-sa-4.0
---
## HOW TO WRANGLING THIS DATASET TO DPO & CHATML FORMAT
```
def return_prompt_and_responses(samples) -> dict[str, str, str]:
return {
"prompt": [
"<|im_start|>user\n" + i + "<|im_end|>\n"
for i in samples["PROMPT"]
],
"chosen": [
"<|im_start|>assistant\n" + j + "<|im_end|>"
for j in samples["CHOSEN"]
],
"rejected": [
"<|im_start|>assistant\n" + k + "<|im_end|>"
for k in samples["REJECTED"]
],
}
dataset = load_dataset(
"Ichsan2895/DPO_ID-Wiki_10kTesting",
)
original_columns = dataset.column_names
dataset.map(
return_prompt_and_responses,
batched=True,
remove_columns=original_columns
)
```
## HOW TO USE DPO
```
dpo_trainer = DPOTrainer(
model, # base model from SFT pipeline
model_ref, # typically a copy of the SFT trained base model
beta=0.1, # temperature hyperparameter of DPO
train_dataset=dataset['train'], # dataset prepared above
tokenizer=tokenizer, # tokenizer
args=training_args, # training arguments e.g. batch size, lr, etc.
)
```
## CITATION
```
@ONLINE{wikidump,
author = "Wikimedia Foundation",
title = "Wikimedia Downloads",
url = "https://dumps.wikimedia.org"
}
@misc{vonwerra2022trl,
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang},
title = {TRL: Transformer Reinforcement Learning},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/trl}}
}
```
许可证：CC-BY-NC-SA-4.0（知识共享署名-非商业性使用-相同方式共享4.0国际协议）
## 如何将本数据集整理为直接偏好优化（Direct Preference Optimization，DPO）与ChatML（对话标记语言）格式
python
def return_prompt_and_responses(samples) -> dict[str, str, str]:
return {
"prompt": [
"<|im_start|>user
" + i + "<|im_end|>
"
for i in samples["PROMPT"]
],
"chosen": [
"<|im_start|>assistant
" + j + "<|im_end|>"
for j in samples["CHOSEN"]
],
"rejected": [
"<|im_start|>assistant
" + k + "<|im_end|>"
for k in samples["REJECTED"]
],
}
dataset = load_dataset(
"Ichsan2895/DPO_ID-Wiki_10kTesting",
)
original_columns = dataset.column_names
dataset.map(
return_prompt_and_responses,
batched=True,
remove_columns=original_columns
)
函数`return_prompt_and_responses`接收样本集作为输入，返回格式为`dict[str, str, str]`的字典，其中：
- `"prompt"`字段：由`<|im_start|>user
`与样本集中`PROMPT`列表的每个元素拼接后再接`<|im_end|>
`组成的字符串列表
- `"chosen"`字段：由`<|im_start|>assistant
`与样本集中`CHOSEN`列表的每个元素拼接后再接`<|im_end|>`组成的字符串列表
- `"rejected"`字段：由`<|im_start|>assistant
`与样本集中`REJECTED`列表的每个元素拼接后再接`<|im_end|>`组成的字符串列表
使用Hugging Face Datasets库的`load_dataset`函数加载标识为`Ichsan2895/DPO_ID-Wiki_10kTesting`的数据集，获取该数据集的原始列名并赋值给`original_columns`变量。随后调用`dataset.map`方法，传入上述`return_prompt_and_responses`函数，启用批处理模式，并移除所有原始列。
## 直接偏好优化（DPO）使用方法
python
dpo_trainer = DPOTrainer(
model, # 来自监督微调（Supervised Fine-Tuning，SFT）流程的基础模型
model_ref, # 通常为经过SFT训练的基础模型的副本
beta=0.1, # DPO的温度超参数
train_dataset=dataset['train'], # 上述预处理完成的训练数据集
tokenizer=tokenizer, # 分词器（Tokenizer）
args=training_args, # 训练参数，例如批次大小、学习率等
)
## 引用格式
bibtex
@ONLINE{wikidump,
author = "Wikimedia基金会",
title = "Wikimedia下载资源",
url = "https://dumps.wikimedia.org"
}
@misc{vonwerra2022trl,
author = {Leandro von Werra、Younes Belkada、Lewis Tunstall、Edward Beeching、Tristan Thrush、Nathan Lambert、Shengyi Huang},
title = {TRL: Transformer强化学习},
year = {2020},
publisher = {GitHub},
journal = {GitHub仓库},
howpublished = {url{https://github.com/huggingface/trl}}
}
