ProlificAI/autoresearch-hitl-annotations

该数据集名为Autoresearch × Prolific HITL dataset，是一个用于偏好学习、人类偏好和直接偏好优化（DPO）研究的注释数据集。它基于一项研究，该研究在DPO任务上运行Karpathy的autoresearch，并使用300名Prolific参与者评估生成的模型。数据集包含两个配置：默认的annotations配置包含1,507行数据，每行代表一个注释者对一对模型响应的成对判断；comment_themes配置包含8行数据，代表通过LLM（Claude模型）聚类生成的注释主题。数据集列包括参与者ID、场景ID、提示ID、模型对、提示文本、模型响应、注释者选择（A、B或平局）、可选自由文本注释等。研究设计涉及5个模型（包括原始模型和四个DPO配方），所有C(5,2)=10个唯一对，50个跨8个类别的通用提示，每个对实例3个独立注释者，最终样本包含来自305名唯一参与者的1,507个成对判断。基础模型为SmolLM2-360M-Instruct，DPO数据集为UltraFeedback。隐私方面，原始参与者ID已被匿名化替换，未收集人口统计数据。数据集可用于评估模型偏好、LLM作为法官方法以及人类注释分析。

This dataset, named Autoresearch × Prolific HITL dataset, is an annotation dataset for preference learning, human preferences, and direct preference optimization (DPO) research. It is based on a study that runs Karpathys autoresearch on a DPO task and evaluates the resulting models with 300 Prolific participants. The dataset includes two configs: the default annotations config contains 1,507 rows, each representing one annotators pairwise judgement on one prompt comparing two model responses; the comment_themes config contains 8 rows, one per LLM-clustered theme over the non-empty annotator comments, generated by Claude. Columns include participant_id, scenario_id, prompt_id, axis, prompt, a_model, b_model, response_a, response_b, choice_canonical (A, B, or tie), comment, etc. The study design involves 5 models (including the original model and four DPO recipes), all C(5,2)=10 unique pairs, 50 general-audience-judgeable prompts across 8 categories, 3 independent annotators per pair-instance, and a final sample of 1,507 pairwise judgements from 305 unique participants. The base model is SmolLM2-360M-Instruct, and the DPO dataset is UltraFeedback. Privacy measures include anonymized participant IDs and no demographic data collection. The dataset is useful for evaluating model preferences, LLM-as-judge methods, and human annotation analysis.