facebook/ExploreToM

ExploreToM是第一个允许大规模生成多样且具有挑战性的理论思维数据的框架，用于鲁棒训练和评估。该方法利用A*搜索算法在自定义领域特定语言上生成复杂的故事结构和新颖、多样但合理的情景，以测试大语言模型的极限。数据集包含10个故事结构，每个故事结构使用50个节点的预算生成，并填充了故事内容。数据集特别适用于测试模型或作为训练数据使用。

ExploreToM is the first framework to allow large-scale generation of diverse and challenging theory of mind data for robust training and evaluation. The framework leverages an A* search algorithm over a custom domain-specific language to produce complex story structures and diverse scenarios to stress test the limits of large language models. The dataset includes 10 story structures across 18 settings, each generated with 50 nodes per story structure. The dataset also includes question-related properties (qprop), story-related properties (sprop), and search parameters (param).