cds-jb/cot-qwen3-8b

CoT Oracle Corpus v5 (Qwen3-8B)数据集是一个复现版本，基于原始ceselder/cot-oracle-corpus-v5数据集，使用Qwen3-8B模型生成。每个问题包含10次独立的rollout（而原始版本为2次），并由Qwen3.5-4B LLM法官进行判断和分类。数据集总共有202,400个条目，对应20,239个唯一问题，聚合cot_correct正确率为92.4%，每个问题多数投票正确率为92.9%。数据来源多样，包括AQUA-RAT、CommonsenseQA、GSM8K、ARC-Easy、ASDiv、MATH和ARC，每个来源的cot_correct率从81.3%到99.0%不等。生成过程采用vLLM 0.20.0引擎，bf16精度，采样参数为temperature=0.6和top_p=0.95，并启用思维链功能，最大新令牌数为16384。分类基于法官对CoT文本和真实答案的评估，分为五个类别：load_bearing（CoT正确但直接回答错误）、both_correct（两者都正确）、both_wrong（两者都错误）、cot_hurt（CoT错误但直接回答正确）和judge_unclear（法官无法判断）。类别分布显示judge_unclear最多（86,027条），both_correct次之（104,066条）。数据集模式在原始基础上扩展，增加了法官判决字段和rollout统计信息，如rollout_majority_correct、rollout_n_correct和rollout_n_total。The CoT Oracle Corpus v5 (Qwen3-8B) dataset is a reproduction of the original ceselder/cot-oracle-corpus-v5, generated using the Qwen3-8B model. It features 10 distinct rollouts per problem (compared to 2 in the original) and is categorized by a Qwen3.5-4B LLM judge. The dataset contains 202,400 entries, corresponding to 20,239 unique problems, with an aggregate cot_correct rate of 92.4% and a per-problem majority-vote correct rate of 92.9%. Data sources include AQUA-RAT, CommonsenseQA, GSM8K, ARC-Easy, ASDiv, MATH, and ARC, with cot_correct rates ranging from 81.3% to 99.0% per source. Generation was performed using the vLLM 0.20.0 engine with bf16 precision, sampling parameters temperature=0.6 and top_p=0.95, and enable_thinking=True, with a maximum of 16,384 new tokens. Categorization is based on the judges evaluation of CoT text and ground truth, resulting in five categories: load_bearing (CoT correct, direct incorrect), both_correct (both correct), both_wrong (both incorrect), cot_hurt (CoT incorrect, direct correct), and judge_unclear (judge unable to decide). The distribution shows judge_unclear as the most frequent (86,027 entries), followed by both_correct (104,066 entries). The schema extends the original with additional fields for judge verdicts and rollout statistics, such as rollout_majority_correct, rollout_n_correct, and rollout_n_total.