everyday-conversations-llama3.1-2k

# Everyday conversations for Smol LLMs finetunings

This dataset contains 2.2k multi-turn conversations generated by [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct). We ask the LLM to generate a simple multi-turn conversation, with 3-4 short exchanges, between a User and an AI Assistant about a certain topic.

The topics are chosen to be simple to understand by smol LLMs and cover everyday topics + elementary science. We include:
- 20 everyday topics with 100 subtopics each
- 43 elementary science topics with 10 subtopics each

All the conversations start with a greeting (`Hi`, `Hello`, `Hey!` or `Hi there`) from the user and a standard assistant answer of `Hello! How can I help you today?`.

You can find the parsed conversations in `messages` column.

## Motivation
This dataset proved to be useful when training small LLMs (in our case the [SmolLM-Instruct](https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966) 135M, 360M and 1.7B models). When training only on the publicly available instructions datasets the models failed to answer basic prompts such as "Hi" (they would bring up other topics) and "Who are you" (failure to realize they are AI assistants).
By including this dataset in the mix, we inject simple everyday behavior for a more user friendly experience.

## Generation
We use [llm-swarm](https://github.com/huggingface/llm-swarm) to generate the conversations, by prompting LLlama-3.1-70B-Instruct with the prompts available in the dataset, using a script similar to [cosmopedia's](https://github.com/huggingface/llm-swarm/tree/main/examples/textbooks). We then parse the completion to extract the conversations.

We notice that the model always uses `Hi` as a greeting, we randomly replace some occurences with `Hello`, `Hey!`, `Hi there` for diversity. By augmenting the training with datasets such as OpenHermes-2.5 and Magpie, the models can also respond correctly to other greeting formats.

# Citation
```
@misc{everydayconversations2024,
author = {Hugging Face},
title = {Everyday Conversations for LLMs},
year = {2024},
howpublished = {\url{https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k}}
}
```

# 面向小型大语言模型（Smol LLMs）微调的日常对话数据集

本数据集包含2200轮由[Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)生成的多轮对话。我们要求大语言模型生成一段3-4轮简短交互的用户与AI助手（AI Assistant）间的多轮对话，对话围绕某一特定主题展开。

所选主题均为小型大语言模型易于理解的内容，涵盖日常话题与基础科学范畴。具体包含：
- 20个日常主题，每个主题下设100个子主题
- 43个基础科学主题，每个主题下设10个子主题

所有对话均以用户的问候（`Hi`、`Hello`、`Hey!`或`Hi there`）开场，助手的标准回复为`Hello! How can I help you today?`。
可在`messages`列中获取解析后的对话内容。

## 研发动机
本数据集在训练小型大语言模型时被证明十分有效（在我们的测试中，针对[SmolLM-Instruct](https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966)的135M、360M与1.7B参数模型进行训练）。若仅使用公开可用的指令数据集进行训练，模型将无法正确响应诸如“Hi”（会跳转至其他话题）与“Who are you”（无法明确自身为AI助手）这类基础提示词。
通过将本数据集纳入训练集，我们可为模型注入贴合日常交互的行为模式，从而提升用户体验的友好度。

## 对话生成流程
我们使用[llm-swarm](https://github.com/huggingface/llm-swarm)工具生成对话：通过向Llama-3.1-70B-Instruct投喂数据集中预设的提示词，并采用与[cosmopedia](https://github.com/huggingface/llm-swarm/tree/main/examples/textbooks)类似的脚本完成生成。随后我们会解析模型输出以提取对话内容。
我们观察到模型默认使用`Hi`作为问候语，因此我们会随机将部分问候替换为`Hello`、`Hey!`与`Hi there`以增加多样性。若结合OpenHermes-2.5与Magpie等数据集进行训练，模型还可正确响应其他格式的问候语。

## 引用格式

@misc{everydayconversations2024,
author = {Hugging Face},
title = {Everyday Conversations for LLMs},
year = {2024},
howpublished = {url{https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k}}
}