hbx/IN3
收藏资源简介：
Intention-in-Interaction (IN3)是一个旨在通过明确的任务模糊性判断和用户意图理解来测试代理交互能力的基准数据集。当前代理基准测试通常忽略了用户意图理解的重要性，IN3通过构建对话记录来训练模型，特别是Mistral-Interact模型，该模型能够判断用户指令的模糊性，主动查询缺失的细节并提供建议，并明确总结详细和清晰的用户意图。数据集包含训练集和测试集，分别包含1,012和95个模糊任务，249和13个清晰任务，250和50个类别，每个任务平均缺失3.57和3.68个细节，每个任务平均有11.39和10.97个选项。Mistral-Interact模型在预测任务模糊性和用户认为必要的缺失细节方面表现最佳，能够有效地进行明确的用户意图总结，并在模糊任务中更合理和友好地询问缺失细节，从而促进对用户隐含意图的更清晰理解。该模型的性能与闭源GPT-4相当，证明了较小规模的模型专家可以在模糊性判断、总结的全面性和交互的友好性等方面接近甚至超过通用大规模模型。
Intention-in-Interaction (IN3) is a benchmark dataset designed to test agent interaction capabilities through explicit task ambiguity judgment and user intention understanding. Current agent benchmark tests often overlook the critical importance of user intention comprehension. IN3 constructs dialogue records for model training, particularly for the Mistral-Interact model, which can assess the ambiguity of user instructions, proactively query for missing details and provide suggestions, and explicitly summarize detailed and clear user intentions. The dataset includes training and test splits, which respectively contain 1,012 and 95 ambiguous tasks, 249 and 13 clear tasks, as well as 250 and 50 task categories. On average, each task in the training and test sets has 3.57 and 3.68 missing details, and 11.39 and 10.97 options respectively. The Mistral-Interact model achieves the best performance in predicting task ambiguity and the missing details deemed necessary by users. It can effectively conduct explicit user intention summarization, and inquire about missing details in a more reasonable and friendly manner for ambiguous tasks, thereby facilitating a clearer understanding of users' implicit intentions. Its performance is comparable to that of closed-source GPT-4, proving that smaller-scale expert models can approach or even outperform general large-scale models in terms of ambiguity judgment, comprehensiveness of summarization, and friendliness of interaction.
数据集卡片：Intention-in-Interaction (IN3)
基本统计信息
| 分割 | 训练 | 测试 |
|---|---|---|
| # 模糊任务 | 1,012 | 95 |
| # 清晰任务 | 249 | 13 |
| # 类别 | 250 | 50 |
| 平均每任务缺失细节数 | 3.57 | 3.68 |
| 平均每任务选项数 | 11.39 | 10.97 |
数据集描述
Intention-in-Interaction (IN3) 是一个旨在通过明确的任务模糊性判断和用户意图理解来测试代理交互能力的基准。为了进一步增强当前代理设计的隐含意图理解能力，我们提出通过与用户的对话来训练一个专门擅长隐含意图理解的模型专家，并将其作为代理设计中的上游模块。
模型特点
- 更好的用户判断理解： 在所有开源模型中，Mistral-Interact 在预测任务模糊性和用户认为必要的缺失细节方面表现最佳。
- 用户意图的全面总结： Mistral-Interact 能够根据详细的用户意图进行明确且全面的总结。
- 增强的模型-用户交互体验： Mistral-Interact 在模糊任务中更合理和友好地询问缺失细节，从而促进对用户隐含意图的更清晰理解。
- 与闭源 GPT-4 相当的性能： 我们证明，较小规模的模型专家可以在模糊性判断、总结的全面性和交互友好性等多个方面接近甚至超过通用的大型模型。
引用
如果您发现此数据集有用，请随意引用我们的论文。
shell @article{cheng2024tell, title={Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents}, author={Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun}, journal={arXiv preprint arXiv:2402.09205}, year={2024} }
