instruction-pilot-outputs-sampling

# Dataset Card for "instruction-pilot-outputs-sampling"

This dataset contains model outputs generated from the human demonstrations provided in [`HuggingFaceH4/instruction-pilot-prompts`](https://huggingface.co/datasets/HuggingFaceH4/instruction-pilot-prompts).

To convert each language model into a dialogue agent, we shortened [Anthropic's HHH prompt](https://gist.github.com/jareddk/2509330f8ef3d787fc5aaac67aab5f11/#file-hhh_prompt-txt) and prepended this to each sample provided to the models:

```
Below is a friendly conversation between a human and an AI assistant. \
The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. \
The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. \
It also tries to avoid giving false or misleading information, and it caveats when it isn’t entirely sure about the right answer. \
That said, the assistant is practical and really does its best, and doesn’t let caution get too much in the way of being useful.

Human: {input}
AI:
```

The reason to shorten the HHH prompt is because it is over 6,000 tokens long, which far exceeds the maximum context size of most open-source language models. For example, Flan-T5 only has a context window of 512 tokens. It is likely that better outputs could be produced for language models with larger context windows, where some dialogue examples can be included in the promopt.

To generate diverse outputs from each models, we used nucleus sampling with `temperature=0` and `top_p=0.9` and set `max_new_tokens=100` (which is about the mean lenght of the Self-Instruct outputs). For each example, 8 generations were produced per model.

# 「instruction-pilot-outputs-sampling」数据集卡片

本数据集包含源自 [`HuggingFaceH4/instruction-pilot-prompts`](https://huggingface.co/datasets/HuggingFaceH4/instruction-pilot-prompts) 数据集所提供的人类演示文本生成的模型输出。

为将各语言模型转换为对话AI智能体（AI Agent），我们对[Anthropic的HHH提示词（HHH prompt）](https://gist.github.com/jareddk/2509330f8ef3d787fc5aaac67aab5f11/#file-hhh_prompt-txt)进行了精简，并将其作为前缀添加至提供给模型的每条样本前：

以下是一段人类与AI助手间的友好对话。
AI力求做到乐于助人、礼貌得体、诚实可靠、谈吐优雅、共情敏锐，且谦逊且学识渊博。
该助手乐于为几乎所有事务提供协助，并将尽全力准确理解用户的需求。
同时，它会避免提供虚假或误导性信息，若无法完全确定正确答案，则会预先说明。
尽管如此，该助手仍兼具实用性与行动力，会全力以赴，不会因过度谨慎而影响服务效用。

人类：{input}
AI:

对HHH提示词进行精简的原因在于其原始长度超过6000个Token（Token），远超多数开源语言模型的最大上下文长度限制。例如，Flan-T5的上下文窗口仅为512个Token。对于拥有更大上下文窗口的语言模型而言，若能在提示词中加入若干对话示例，或可生成更优质的输出结果。

为从各模型生成多样化输出，我们采用了核采样（nucleus sampling）策略，设置`temperature=0`、`top_p=0.9`，并将`max_new_tokens`设为100（该数值大致对应自指令（Self-Instruct）输出的平均长度）。针对每个示例，每个模型均生成了8条回复。