jizej/Competence-Based-Evaluation

该数据集名为基于能力的评估（不变性基准），旨在测试语言模型在逻辑排序问题的语义等价重新表述下是否给出相同答案。数据集包含评估和监督微调（SFT）两个主要部分。评估部分包括多个配置（如eval_pos、eval_pos_largeN、eval_depth），每个配置包含original和equivalent配对记录，这些记录描述相同的底层排序但使用不同的表面措辞（例如，Alice在Bob前面与Bob在Alice后面）。通过比较模型在配对记录上的答案一致性来测量不变性，并根据真实布尔值测量准确性。SFT部分为聊天格式（messages字段），适用于监督微调训练，包含sft_full和sft_noleak等配置，使用与评估集不同的事实无关关系生成，并排除了评估中使用的关系以确保分布外泛化。数据集完全由合成数据生成，通过确定性程序生成器构建，不涉及人类标注或模型生成。数据源包括Wikidata SPARQL查询、Wikipedia API和人工整理的实体列表（如人名、城市、动物等）。数据集主要用于评估语言模型在逻辑排序问题中的答案不变性，并支持微调研究。局限性包括仅测试线性排序的传递闭包、表面形式多样性有限、仅限英语、输出空间为是/否、以及可能的长上下文混淆等。The dataset is named Competence-Based Evaluation (Invariance Benchmark) and is designed to test whether language models give the same answer to semantically equivalent reformulations of logical-ordering questions. It consists of two main parts: evaluation and supervised fine-tuning (SFT). The evaluation part includes multiple configurations (e.g., eval_pos, eval_pos_largeN, eval_depth), each containing paired original and equivalent records that describe the same underlying ordering but with different surface phrasings (e.g., Alice in front of Bob vs. Bob behind Alice). Invariance is measured by the agreement between the models answers on paired records, and accuracy is measured against the ground-truth boolean. The SFT part is in chat format (messages field) and is suitable for supervised fine-tuning training, including configurations like sft_full and sft_noleak, which are generated using fact-agnostic relations different from the evaluation set and exclude the relations used in evaluation to ensure out-of-distribution generalization. The dataset is fully synthetic, generated by a deterministic procedural generator, with no human annotation or model-based generation involved. Data sources include Wikidata SPARQL queries, Wikipedia API, and manually curated entity lists (e.g., names, cities, animals). The primary use cases are evaluating answer-level invariance in language models for logical ordering and supporting fine-tuning research. Limitations include testing only transitive closure over linear orderings, limited surface-form diversity, English-only prompts, yes/no output space, and potential long-context confounds.