ytu-ce-cosmos/tubitak-science-olympiad-tr

--- language: - tr task_categories: - question-answering - multiple-choice - visual-question-answering - text-generation pretty_name: TUBITAK Science Olympiad Dataset size_categories: - 1K<n<10K license: cc-by-4.0 dataset_info: features: - name: id dtype: string - name: subject dtype: string - name: year dtype: int64 - name: stage dtype: int64 - name: question_number dtype: int64 - name: question_image dtype: image - name: solution_image dtype: image - name: question_latex dtype: string - name: solution_latex dtype: string - name: has_solution dtype: bool - name: has_figure dtype: bool - name: has_choices dtype: bool - name: choice_values dtype: string - name: has_answer dtype: bool - name: answer_letter dtype: string - name: answer_value dtype: string splits: - name: bilgisayar num_bytes: 178220803.0 num_examples: 863 - name: fizik num_bytes: 106735713.0 num_examples: 332 - name: matematik num_bytes: 129297914.0 num_examples: 671 - name: ortaokul_bilgisayar num_bytes: 38764774.0 num_examples: 233 - name: ortaokul_matematik num_bytes: 87348043.0 num_examples: 599 download_size: 528873937 dataset_size: 540367247.0 configs: - config_name: default data_files: - split: bilgisayar path: data/bilgisayar-* - split: fizik path: data/fizik-* - split: matematik path: data/matematik-* - split: ortaokul_bilgisayar path: data/ortaokul_bilgisayar-* - split: ortaokul_matematik path: data/ortaokul_matematik-* --- # TUBITAK Science Olympiad Dataset This dataset contains multiple-choice and open-ended scientific questions sourced from the TUBITAK (The Scientific and Technological Research Council of Turkey) Science Olympiads spanning various years. It is intended to serve as a benchmark for evaluating the advanced analytical, mathematical, and computational reasoning capabilities of Large Language Models (LLMs) in the Turkish language. The dataset comprises approximately 2700 problems across five domains: Computer Science, Physics, Mathematics, Middle School Computer Science, and Middle School Mathematics. The raw problems have been formatted, OCR processed (using `deepseek-ai/DeepSeek-OCR-2`), and augmented with structural rules to test multi-step reasoning. ## Dataset Structure Each entry in the dataset represents a specific problem from the competition stages (typically Stage 1). - **id**: Unique identifier of the problem (e.g., Matematik_2024_1.Asama_1). - **subject**: Science domain (Matematik, Fizik, Bilgisayar, Ortaokul Matematik, vb.). - **year**: The year of the examination. - **stage**: Examination stage (1 or 2). Note: Computer Science and Physics contain only Stage 1 questions. - **question_number**: The specific problem number within the exam booklet. - **question_image**: The primary image associated with the question. - **solution_image**: Link to the solution image (if any). - **question_latex**: The textual representation of the problem (includes LaTeX formulations where applicable). - **solution_latex**: LaTeX formatted solution text (if any). - **has_solution**: Indicates whether the problem has a solution. - **has_figure**: Boolean flag indicating if the problem essentially relies on visual context (accuracy is not 100%). - **has_choices**: Indicates whether the question is multiple-choice or open-ended. - **choice_values**: Array of multi-choice options (A, B, C, D, E). - **has_answer**: Indicates whether the problem has an answer. - **answer_letter**: The correct choice letter. - **answer_value**: The actual content of the correct choice. ## Important Characteristics & Limitations - **Visual Context:** Visuals within questions are marked as [IMAGE]. For problems sharing a common block of text or context, the explanatory text/image is embedded on top of the question image of the respective problem. The context format traditionally ends with `\n---\n`. - **Cancellations:** Most cancelled questions from the official exams were skipped; however, recoverable ones were preserved (e.g., Middle School Computer-2020-Stage1-Booklet A-8 and 9 vs skipped Computer-2014-Stage1-28,30,31). - **Reference Links:** Solutions to questions that strictly reference the previous problem have been largely modified to be standalone, but perfection is not guaranteed (see Computer-2020-Stage1-21 and 23). - **Code Excerpts:** In Computer Science branches, the last 10-15 questions are typically C programming tasks formatted heavily in LaTeX. While recent years (e.g., 2025) might have these converted directly to images, older ones (e.g., 2024) do not always have an briefing image. Furthermore, any raw C code present in questions is wrapped within standard markdown c bracket blocks for clarity. - **AI Intervention:** Please note that artificial intelligence (specifically OCR models) was utilized during the creation and structuring of this dataset, which carries a limited accuracy rate for complex LaTeX rendering. ## Usage This dataset is particularly useful for: - **Benchmarking:** Testing LLMs on demanding, multi-step scientific reasoning tasks in non-English contexts. - **Multimodal Evaluation:** Correlating highly visual problem spaces (like the Physics branch) with text-only analytical capabilities. - **Chain-of-Thought (CoT) Capabilities:** Eliciting formal proofs and deep understanding in mathematics, kinematics, and logic/code tracing. ## LLM Performance Evaluation / Benchmark The most recent 2 years of Stage 1 questions for all active branches were evaluated using a strict single-prompt, Chain-of-Thought approach. Models were tasked to reason step-by-step and strictly output the final choice letter. *(Cancelled ("IPTAL") problems were excluded from Accuracy calculations)* | Model | Total Pass | Total Fail | Cancelled (Ignored) | Accuracy | |:---|:---:|:---:|:---:|:---:| | **Gemini 3.1 Pro** | 326 | 2 | 10 | **99.39%** | | **Qwen3.5-397B-A17B + Thinking** | 319 | 10 | 9 | **96.96%** | ### Branch-Specific Overview | Branch | Qwen Pass | Qwen Fail | Qwen Acc. | Gemini Pass | Gemini Fail | Gemini Acc. | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | Computer | 148 | 6 | 96.10% | 152 | 1 | 99.35% | | Mathematics | 123 | 4 | 96.85% | 126 | 1 | 99.21% | | Physics | 48 | 0 | 100.00% | 48 | 0 | 100.00% | ## Source & License The original problems are sourced from the national science olympiads organized by TUBITAK (The Scientific and Technological Research Council of Turkey). This formalized dataset is provided for research and educational purposes under the **CC BY 4.0** license. Necessary permissions have been acquired from TUBITAK by the research team for publishing this derived benchmark. ## Contact COSMOS AI Research Group Yildiz Technical University Computer Engineering Department https://cosmos.yildiz.edu.tr/ [e-mail]: tr task_categories: - 问答任务 - 多项选择任务 - 视觉问答任务 - 文本生成任务 pretty_name: TUBITAK科学奥林匹克数据集 size_categories: - 1000<n<10000 license: CC BY 4.0 dataset_info: features: - name: id dtype: 字符串 - name: subject dtype: 字符串 - name: year dtype: 64位整数 - name: stage dtype: 64位整数 - name: question_number dtype: 64位整数 - name: question_image dtype: 图片 - name: solution_image dtype: 图片 - name: question_latex dtype: 字符串 - name: solution_latex dtype: 字符串 - name: has_solution dtype: 布尔值 - name: has_figure dtype: 布尔值 - name: has_choices dtype: 布尔值 - name: choice_values dtype: 字符串 - name: has_answer dtype: 布尔值 - name: answer_letter dtype: 字符串 - name: answer_value dtype: 字符串 splits: - name: bilgisayar num_bytes: 178220803.0 num_examples: 863 - name: fizik num_bytes: 106735713.0 num_examples: 332 - name: matematik num_bytes: 129297914.0 num_examples: 671 - name: ortaokul_bilgisayar num_bytes: 38764774.0 num_examples: 233 - name: ortaokul_matematik num_bytes: 87348043.0 num_examples: 599 download_size: 528873937 dataset_size: 540367247.0 configs: - config_name: default data_files: - split: bilgisayar path: data/bilgisayar-* - split: fizik path: data/fizik-* - split: matematik path: data/matematik-* - split: ortaokul_bilgisayar path: data/ortaokul_bilgisayar-* - split: ortaokul_matematik path: data/ortaokul_matematik-* # TUBITAK科学奥林匹克数据集 本数据集收录源自土耳其科学和技术研究理事会（TUBITAK）历年科学奥林匹克竞赛的多项选择及开放式科学试题，旨在作为基准数据集，用于评估大语言模型（LLM）在土耳其语语境下的高级分析、数学与计算推理能力。 数据集涵盖计算机科学、物理学、数学、初中计算机科学、初中数学五个领域，共计约2700道试题。原始试题已完成格式标准化处理，并使用`deepseek-ai/DeepSeek-OCR-2`进行光学字符识别（OCR），同时结合结构化规则进行增强，以测试模型的多步推理能力。 ## 数据集结构 数据集中的每个条目对应竞赛某一阶段（通常为第一阶段）的一道具体试题。 - **id**：试题的唯一标识符（例如：Matematik_2024_1.Asama_1）。 - **subject**：学科领域（如数学、物理、计算机、初中数学等）。 - **year**：考试年份。 - **stage**：竞赛阶段（1或2）。注：计算机科学与物理领域仅包含第一阶段试题。 - **question_number**：试卷内的具体试题编号。 - **question_image**：与试题关联的主图片。 - **solution_image**：解析图片的链接（若存在）。 - **question_latex**：试题的文本表示形式（若适用则包含LaTeX公式）。 - **solution_latex**：LaTeX格式的解析文本（若存在）。 - **has_solution**：标识试题是否包含解析。 - **has_figure**：布尔标记，指示试题是否依赖视觉上下文（准确率并非100%）。 - **has_choices**：标识试题为选择题还是开放式题。 - **choice_values**：多项选择的选项集合（A、B、C、D、E）。 - **has_answer**：标识试题是否包含答案。 - **answer_letter**：正确选项的字母标识。 - **answer_value**：正确选项的具体内容。 ## 重要特征与局限性 - **视觉上下文**：试题中的视觉元素标记为`[IMAGE]`。对于共享同一文本块或上下文的试题，解释性文本/图片会嵌入对应试题的试题图片顶部。上下文格式通常以` --- `结尾。 - **作废试题**：官方考试中的多数作废试题已被跳过，但可恢复的试题予以保留（例如：初中计算机-2020-第一阶段-试卷A-第8、9题，与已跳过的计算机-2014-第一阶段-第28、30、31题对比）。 - **参考链接**：严格引用前一道试题的解析已大部分修改为独立内容，但无法保证完全完善（例如计算机-2020-第一阶段-第21、23题）。 - **代码片段**：在计算机科学领域，最后10-15道试题通常为以LaTeX格式排版的C语言编程任务。近年（如2025年）的此类试题可能直接转换为图片，而旧试题（如2024年）未必附带导读图片。此外，试题中出现的原始C语言代码会用标准markdown代码块包裹以提升可读性。 - **AI干预**：请注意，本数据集的创建与结构化过程中使用了人工智能工具（特指OCR模型），复杂LaTeX渲染的准确率存在一定局限。 ## 使用场景 本数据集尤其适用于： - **基准测试**：在非英语语境下，测试大语言模型处理高难度多步科学推理任务的能力。 - **多模态评估**：将物理等高度依赖视觉的试题领域与纯文本分析能力进行关联评估。 - **思维链（Chain-of-Thought，CoT）能力**：激发数学、运动学、逻辑与代码追踪领域的形式化证明与深度理解能力。 ## 大语言模型性能评估/基准测试 针对所有活跃领域的最近两年第一阶段试题，采用严格单提示思维链方法进行评估。模型需逐步推理，并严格输出最终选项字母。 （作废（“IPTAL”）试题未纳入准确率计算） | 模型 | 总通过数 | 总失败数 | 作废（忽略）数 | 准确率 | |:---|:---:|:---:|:---:|:---:| | **Gemini 3.1 Pro** | 326 | 2 | 10 | **99.39%** | | **Qwen3.5-397B-A17B + 思维链** | 319 | 10 | 9 | **96.96%** | ### 分领域概览 | 领域 | Qwen通过数 | Qwen失败数 | Qwen准确率 | Gemini通过数 | Gemini失败数 | Gemini准确率 | |:---|:---:|:---:|:---:|:---:|:---:|:---:| | 计算机 | 148 | 6 | 96.10% | 152 | 1 | 99.35% | | 数学 | 123 | 4 | 96.85% | 126 | 1 | 99.21% | | 物理 | 48 | 0 | 100.00% | 48 | 0 | 100.00% | ## 来源与许可协议 原始试题源自土耳其科学和技术研究理事会（TUBITAK）组织的全国科学奥林匹克竞赛。 本标准化数据集以**CC BY 4.0**许可协议发布，仅供研究与教育用途。研究团队已从TUBITAK获取必要许可，方可发布此衍生基准数据集。 ## 联系方式 COSMOS人工智能研究团队 伊迪尔技术大学计算机工程系 https://cosmos.yildiz.edu.tr/ [e-mail]