POUncertainty-Benchmarking

POUncertainty-Benchmarking数据集由国立台湾大学创建，旨在评估大型语言模型（LLMs）在提示优化中的不确定性。该数据集包含大量树状推理轨迹，通过随机扰动和采样生成，涵盖了回答不确定性、正确性不确定性、偶然不确定性和认知不确定性等多种类型。数据集的创建过程涉及对输入问题的随机扰动、模型输出的随机采样以及不确定性计算。该数据集主要应用于提示优化任务，旨在提高多步骤推理任务中模型的不确定性估计精度。POUncertainty-Benchmarking dataset was developed by National Taiwan University to evaluate the uncertainty of large language models (LLMs) in the context of prompt optimization. This dataset comprises a large number of tree-structured reasoning trajectories generated via stochastic perturbation and sampling, covering multiple categories of uncertainty including answer uncertainty, correctness uncertainty, aleatoric uncertainty, and epistemic uncertainty. The construction process of this dataset involves random perturbation of input queries, stochastic sampling of model outputs, and uncertainty quantification. This benchmark is primarily applied to prompt optimization tasks, aiming to improve the precision of model uncertainty estimation in multi-step reasoning scenarios.