pkuHaowei/kumo-hard

KUMO是一个用于系统评估大型语言模型复杂推理能力的新型基准，通过程序生成的推理游戏进行。数据集包含了100个自动生成的示例领域，覆盖计算机科学、生物学、艺术等多个类别，每个领域包含大约50个真实和30个动作。数据集的结构包括知识库、动作列表、真实列表、最优动作数量、观察值和地面真实值，分为MedicalEnv、EducationEnv、MusicEnv、FantasyEnv和ChemicalEnv五个环境，每个环境有50个示例。

KUMO is a novel benchmark for systematically evaluating complex reasoning capabilities in Large Language Models (LLMs) through procedurally generated reasoning games. The dataset includes 100 autogenerated exemplar domains across categories such as Computer Science, Biology, Art, and more, with each domain containing about 50 truths and 30 actions. The dataset structure includes a knowledge book, action list, truth list, optimal action count, observation, and ground truth, divided into five environments: MedicalEnv, EducationEnv, MusicEnv, FantasyEnv, and ChemicalEnv, with each environment having 50 examples.