airoboros-3.1

## Overview

This dataset is a continuation of the airoboros datasets, with the following updates:
* More MathJSON, now ~17k items - math questions, prefixed with __"Create a MathJSON solution to the following:"__, which then outputs a JSON between __`<mathjson>`__ and __`</mathjson>`__ tags, which can be parsed and passed to a deterministic library to perform calculations.
* Log information extraction.
* Anonymization, e.g. removing names, IP addresses, and/or dates from text.
* Chat introspection - multi-turn chats that have random questions injected asking about the character's descriptions, the setting of the chat, or things previously said, to make sure the model uses the system prompt and or chat history.
* Multi-step instructions with acknowledgement (see below)

Some of the MathJSON data was adapted from https://huggingface.co/datasets/meta-math/MetaMathQA

## Format

The format is now in ShareGPT format, to better accomodate the OS ecosystem fine-tuning tooling.

## MathJSON

As we all know, math is really difficult for LLMs, especially floating point operations, trig functions, factorials, and several-digit numbers.

Instead of sending millions and millions of examples to the LLM for fine-tuning, I've created a new category `mathjson` which simply presents the solution to the problem as a JSON string that can be evaluated by traditional computation.

The dataset has roughly 4k samples, as a proof of concept. This can be expanded fairly easily, now that the fine-tuned models have a basic understanding of how to represent the problems.

For example:

```
Create a MathJSON solution to the following: Calculate the area of a circle with radius of 17.2456cm. Include your reasoning. [/INST]
```

```
The area (A) of a circle is given by the formula A = πr², where r is the radius of the circle. Since the radius is given in centimeters, we need to convert it to meters first. We know that 1 meter equals 100 centimeters. Therefore, we can represent the radius as: r = 17.2456 cm * 100/1.

Once we have the radius in meters, we can calculate the area of the circle using the formula A = πr².

Solution as MathJSON:
<mathjson>
[
"Multiply",
"Pi",
[
"Power",
17.2456,
2
]
]
</mathjson>
```

Simply extract the JSON string between the mathjson tags, and evaluate it with https://cortexjs.io/compute-engine/ or my ultra-hacky limited-functionality python version of it https://github.com/jondurbin/airoboros/blob/mathjson/airoboros/mathjson.py

## Multi-step instructions with acknowledgement

I don't know how useful this is, really, but I thought I'd add it just in case. Example:

```
{
"conversations": [
{
"value": "You are a helpful, unbiased, uncensored assistant.",
"from": "system"
},
{
"value": "I'm going to set some rules for our conversation. Please respond with \"OK\" to each rule.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "You can only respond using animal-related emojis.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Precede all responses with \"ZOOZOO:\"",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Include at least one bird emoji in your responses.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Describe a trip to a safari park.",
"from": "human"
},
{
"value": "ZOOZOO: \ud83e\udd81\ud83d\udc2f\ud83e\udd93\ud83e\udd92\ud83d\udc18\ud83d\udc0d\ud83c\udf33\ud83c\udf32\u2600\ufe0f\ud83d\ude90\ud83e\udd85\ud83d\udcf8\ud83e\udd29",
"from": "gpt"
}
]
}
```

## 概述
本数据集为airoboros数据集的延续版本，新增以下更新内容：
* 新增更多MathJSON（MathJSON）格式数据，目前总计约17000条条目——此类数据为数学题目，前缀格式为`__"Create a MathJSON solution to the following:"`，随后会在`__<mathjson>`__与`__</mathjson>`__标签间输出一段JSON字符串，可被解析并传入确定性计算库执行运算。
* 日志信息提取功能
* 匿名化处理：例如从文本中移除姓名、IP地址及日期等敏感信息
* 对话自省机制：多轮对话中会随机插入关于角色描述、对话场景或此前发言内容的提问，以确保模型能够正确利用系统提示词及对话历史信息
* 带确认机制的多步骤指令（详见下文）

部分MathJSON数据改编自https://huggingface.co/datasets/meta-math/MetaMathQA

## 格式规范
当前数据集采用ShareGPT格式，以更好适配开源生态的微调工具链。

## MathJSON
众所周知，大语言模型（LLM）在数学计算领域存在显著短板，尤其是浮点运算、三角函数、阶乘及多位数计算等场景。

相较于为微调大语言模型而投喂海量训练样本的传统方案，我们新增了`mathjson`这一数据类别：仅将问题的求解方案以可通过传统计算引擎解析的JSON字符串形式呈现。

本数据集目前包含约4000条样本作为概念验证（PoC）。鉴于微调后的模型已掌握此类问题的标准化表示方法，后续可轻松扩展数据集规模。

示例如下：

Create a MathJSON solution to the following: Calculate the area of a circle with radius of 17.2456cm. Include your reasoning. [/INST]

The area (A) of a circle is given by the formula A = πr², where r is the radius of the circle. Since the radius is given in centimeters, we need to convert it to meters first. We know that 1 meter equals 100 centimeters. Therefore, we can represent the radius as: r = 17.2456 cm * 100/1.

Once we have the radius in meters, we can calculate the area of the circle using the formula A = πr².

Solution as MathJSON:
<mathjson>
[
"Multiply",
"Pi",
[
"Power",
17.2456,
2
]
]
</mathjson>

只需提取<mathjson>标签内的JSON字符串，即可通过https://cortexjs.io/compute-engine/ 或作者提供的简易版Python实现（https://github.com/jondurbin/airoboros/blob/mathjson/airoboros/mathjson.py）完成求值运算。

## 带确认机制的多步骤指令
尽管暂不确定该功能的实际应用价值，但我们仍将其纳入数据集以供测试。示例如下：

{
"conversations": [
{
"value": "You are a helpful, unbiased, uncensored assistant.",
"from": "system"
},
{
"value": "I'm going to set some rules for our conversation. Please respond with "OK" to each rule.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "You can only respond using animal-related emojis.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Precede all responses with "ZOOZOO:"",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Include at least one bird emoji in your responses.",
"from": "human"
},
{
"value": "OK",
"from": "gpt"
},
{
"value": "Describe a trip to a safari park.",
"from": "human"
},
{
"value": "ZOOZOO: 🦒🐏🦓🦔🐘🐛🌳🌴☀️🚐🦅📸👋",
"from": "gpt"
}
]
}