SkillFactory/BF_EVAL-cd3args-Qwen2.5-1.5B-Instruct-SkillFactory-RL

--- license: mit task_categories: - text-generation language: - en tags: - evaluation - skill-factory --- These datasets are exactly like the Evaluation datasets except the model_responses array are budget forcing rounds. So the first response is at a maximum total context length of 4k, the second response (2nd index in the array) is a continuation of that last response up to a total of 8,192 tokens. # Column Details | Column | Description | |--------|-------------| | `question` | The question we want the model to answer | | `answer` | The string answer | | `task` | The name of the task the row belongs to | | `prompt` | The prompt we will feed into the model to solve the question | | `model_responses` | An array of strings that the model generated to answer the prompt (usually size of 4 or 34 depending on the evaluation task) | | `model_responses__eval_is_correct` | An array aligned with `model_responses` containing booleans: `True` when the response was correct, `False` when incorrect or no answer was found | | `model_responses__eval_extracted_answers` | An array aligned with `model_responses` containing the extracted answer strings from each response (usually the last answer in `<answer>` tags) | | `model_responses__internal_answers__eval_is_correct` | An array aligned with `model_responses` where each value is an array of booleans for the correctness of intermediate answers within a trace | | `model_responses__internal_answers__eval_extracted_answers` | Similar to `model_responses__eval_extracted_answers` but for internal/intermediate answers | | `all_other_columns` | A catch-all column for additional task-dependent information (e.g., for countdown: target number and arguments) | | `metadata` | Metadata about the question (alternative location for task-specific data like countdown target/arguments) | | `prompt__metadata` | Metadata for the vLLM network request including URL and generation parameters. We used a customized [Curator](https://github.com/bespokelabsai/curator) to send raw text to `/completion` instead of `/chat/completion` for warm-start prompts and budget forcing | | `model_responses__metadata` | Metadata returned from the vLLM request | **Additional task-specific columns:** `answer_index`, `answer_key`, `choices`, `id`, `difficulty`, `domain`, `evaluation_type`, `expected_answer_format`, `original_answer`, `source`, `task_type`, `variant`, `acronym`, `formed_acronym`, `word_count`, `words`, `length`, `letters`--- 许可证：MIT协议 任务类别： - 文本生成（text-generation） 语言： - 英语（en） 标签： - 评测（evaluation） - 技能工厂（skill-factory） --- 本数据集与评测数据集完全一致，仅`model_responses`数组对应的是预算强制（budget forcing）轮次。 首个响应的最大总上下文长度为4096（即4k）个令牌（Token），数组中第二个响应（索引为2）则是前一响应的延续，总长度上限为8192个令牌（Token）。 # 列详情 | 列名 | 说明 | |--------|-------------| | `question` | 需模型作答的问题 | | `answer` | 字符串形式的标准答案 | | `task` | 当前数据行所属的任务名称 | | `prompt` | 用于输入模型以求解该问题的提示词 | | `model_responses` | 模型为响应该提示词生成的字符串数组（根据评测任务的不同，数组长度通常为4或34） | | `model_responses__eval_is_correct` | 与`model_responses`对齐的布尔值数组：当模型响应正确时为`True`，响应错误或未找到有效答案时为`False` | | `model_responses__eval_extracted_answers` | 与`model_responses`对齐的数组，存储从每个模型响应中提取的答案字符串（通常为`<answer>`标签内的最后一个答案） | | `model_responses__internal_answers__eval_is_correct` | 与`model_responses`对齐的数组，其中每个元素为子布尔数组，用于表示推理轨迹内中间答案的正确性 | | `model_responses__internal_answers__eval_extracted_answers` | 与`model_responses__eval_extracted_answers`类似，但针对内部/中间答案 | | `all_other_columns` | 通用列，用于存储与任务相关的额外信息（例如倒计时任务中的目标数字与参数） | | `metadata` | 问题相关的元数据（可替代存储任务专属信息，如倒计时任务的目标数字与参数） | | `prompt__metadata` | vLLM网络请求的元数据，包含请求地址与生成参数。我们使用定制化的[Curator（https://github.com/bespokelabsai/curator）](https://github.com/bespokelabsai/curator)，将原始文本发送至`/completion`接口而非`/chat/completion`，以适配预热提示词与预算强制任务 | | `model_responses__metadata` | vLLM请求返回的元数据 | **额外任务专属列：** `answer_index`、`answer_key`、`choices`、`id`、`difficulty`、`domain`、`evaluation_type`、`expected_answer_format`、`original_answer`、`source`、`task_type`、`variant`、`acronym`、`formed_acronym`、`word_count`、`words`、`length`、`letters`