glebmikheev123/generation-instability

该数据集是用于研究语言模型推理不稳定性的主要问题库，包含来自多个推理环境（如GSM8K、MATH500、HotpotQA、BBH、MMLU-STEM）的标准化基准问题。它旨在支持两种稳定性研究：类型1（运行间/输出稳定性），即重复LLM生成以测量最终答案在不同运行中的一致性和正确性；类型2（生成内/动态稳定性），即使用令牌级对数概率测量单次生成过程中的熵、连续步骤分布偏移和峰值不稳定性。数据集总共有2,500条记录，涵盖数学问题、文本生成和多选问答等多种格式，并提供了标准化的数据结构，但不包含模型输出、重复运行数据或稳定性指标。This dataset is the primary problem bank for studying reasoning instability in language models. It contains normalized benchmark problems from several reasoning environments (e.g., GSM8K, MATH500, HotpotQA, BBH, MMLU-STEM). It is intended to support two types of stability studies: Type 1 (inter-run/output stability), involving repeated LLM generations to measure consistency and correctness of final answers across runs; and Type 2 (intra-generation/dynamic stability), involving token-level logprobs to measure entropy, consecutive-step distribution shift, and peak instability during a single generation. The dataset comprises 2,500 records in total, covering various formats such as math problems, short text answers, and multiple-choice questions, with a normalized schema, but does not include model outputs, repeated runs, or instability metrics.