marin-community/synth-reasoning-arrow-icl-ppl

该数据集名为“合成推理箭头ICL PPL”，是一个用于评估语言模型在上下文学习（ICL）中推理能力的数据集，特别关注困惑度（PPL）比较。数据集包含多个配置，每个配置有1000个验证行，数据格式包括监督的输入（input）和目标（target）字段。提示格式为`arrow_5shot_icl_v1`，每个提示包含五个已解决的上下文示例，格式为“问题 => 答案”，后跟一个待解决的保留问题，以“=> ”结束。这种设计避免了使用“User:”/“Assistant:”角色标签，便于基础模型的困惑度评估，同时实现仅目标评分。数据集覆盖多种推理任务，包括算术（如stepmath_arithmetic）、代数（如stepmath_algebra_linear_equation）、算法（如native_binary_search、native_dijkstra_shortest_path、native_knapsack_01_dp等）以及CLRS风格的算法问题（如clrs_style_clrs_bfs）。数据集生成于2026-05-11，来源于`marin-community/synth-bootstrap-trial`，采用Apache 2.0许可证。This dataset is named "Synthetic Reasoning Arrow ICL PPL", which is designed to evaluate the reasoning capabilities of language models during in-context learning (ICL), with a specific focus on perplexity (PPL) comparisons. The dataset includes multiple configurations, each containing 1000 validation rows. The data format features supervised input and target fields. The prompt format is `arrow_5shot_icl_v1`. Each prompt consists of five solved contextual examples formatted as "Question => Answer", followed by a held-out question to be solved, ending with "=> ". This design avoids using "User:"/"Assistant:" role tags, facilitating perplexity evaluation for base models while enabling target-only scoring. The dataset covers a diverse range of reasoning tasks, including arithmetic (e.g., stepmath_arithmetic), algebra (e.g., stepmath_algebra_linear_equation), algorithms (e.g., native_binary_search, native_dijkstra_shortest_path, native_knapsack_01_dp, etc.), and CLRS-style algorithmic problems (e.g., clrs_style_clrs_bfs). The dataset was generated on 2026-05-11, sourced from `marin-community/synth-bootstrap-trial`, and is licensed under Apache 2.0.