BossCrafts/CodeX-2M-Thinking

---
license: apache-2.0
pretty_name: CodeX-5M-Thinking
dataset_name: XenArcAI/CodeX-5M-Thinking
size_categories:
- 1M<n<10M
language:
- en
task_categories:
- text-generation
- question-answering
tags:
- Coding
- Code
- CodeX
- XenArcAI
- LLM-training
- synthetic
- curated
- benchmark
- reasoning-dataset
- artifact
annotations_creators:
- machine-generated
- expert-verified
source_datasets:
- XenArcAI internal synthetic generation
multilinguality:
- monolingual
---
# XenArcAI
---

<p align="center">
<img
src="https://cdn-uploads.huggingface.co/production/uploads/677fcdf29b9a9863eba3f29f/ZP4YDDIRewH5M-jKmE4Rt.png"
alt="CodeX Banner"
width="70%"
style="border-radius:15px;"
/>

> Note: This dataset is part of the lineup CodeX by XenArcAI. You can get lots of datasets in this same lineup, with the main focus on providing very high-quality datasets for model training and fine-tuning.

This dataset is fully synthetic, curated from high-quality public sources and enhanced with synthetic data generated using both closed and open-source models. It serves as a strong foundation for instruction-based model tuning and fine-tuning, offering one of the most refined and extensive corpora available for coding tasks with reasoning.

### Key Features
- **Scale**: 2 million examples of highly curated coding data
- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks

## Dataset Overview
**CodeX-2M-Thinking** is a meticulously curated coding dataset designed specifically for instruction-based model tuning and fine-tuning of existing models with enhanced code generation and reasoning capabilities. This fully synthetic dataset represents a large and comprehensively filtered corpus of coding data on the Hugging Face platform, emphasizing a thinking approach with step-by-step reasoning for deeper model training.

## How to Use?
```bash
pip install -U datasets fsspec
```
```python
from datasets import load_dataset
dataset = load_dataset("XenArcAI/CodeX-2M-Thinking")
```

### Key Features
- **Scale**: 2 million examples of highly curated coding data
- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks

## Data Curation Process
This dataset has been carefully constructed through a fully synthetic approach, selectively generating and merging examples to enrich the overall dataset for generation models.

### Data Sources
- **High-Quality Existing Datasets**: Curated from multiple premium coding datasets available online (e.g., from NVIDIA and XenArcAI's internal collections)
- **Synthetic Generation**: Fully generated using both closed-source and open-source language models (XenArcAI)
- **Expert Validation**: Human-verified code solutions, reasoning, and implementations (XenArcAI)

### Filtering Pipeline
Our rigorous filtering process includes open and closed-source filtering techniques, ensuring only the highest-quality examples are retained:
1. **Deduplication**: Removal of duplicate problems and code solutions
2. **Normalization**: Code formatting standardization and syntax cleanup
3. **Stopword Processing**: Intelligent removal of non-essential comments or boilerplate
4. **Quality Scoring**: Multi-dimensional quality assessment using metrics like code complexity, readability, and efficiency
5. **Ranking-Based Filtering**: Advanced ranking algorithms to prioritize top-tier examples based on relevance, novelty, and utility
6. **Expert Selections**: Manual curation by coding experts to select exemplary samples
7. **Answer Verification**: Automated testing and execution validation using frameworks like pytest or unit tests
8. **Content Filtering**: Removal of inappropriate, outdated, or incorrect code
9. **Diversity Balancing**: Ensuring balanced representation across languages and domains through algorithmic sampling

### Problem Complexity Distribution
- **Basic Level** (30%): Fundamental programming concepts, simple syntax, and basic operations
- **Intermediate Level** (30%): Multi-function problems requiring modular code and basic algorithms
- **Advanced Level** (40%): Complex challenges involving data structures, optimization, and system design

### Programming Domains Covered
- Algorithms and Data Structures
- Web Development and Frameworks
- Machine Learning and AI Implementations
- System Programming and Operating Systems
- Database Management and SQL/NoSQL
- Software Engineering Best Practices
- Competitive Programming Problems
> Note: Domains are for reference only. The actual data is very diverse and covers more domains than stated. The actual data includes more complex and high-level questions than stated, spanning multiple programming languages such as Python, Java, C++, JavaScript, and others.

## Use Cases
- **Fine-tuning** code generation and reasoning capabilities in language models
- **Training** instruction-following models with a coding and reasoning focus
- **Benchmarking** model performance on coding tasks, problem-solving, and logical reasoning
- **Research** in AI-assisted programming, automated code completion, and explainable AI
- **Educational** applications requiring step-by-step code explanations and reasoning

## Dataset Format
Each example contains:
- **Problem Statement**: Clear coding challenge or task description
- **Step-by-Step Solution**: Detailed reasoning process
- **Code Solution**: Final executable code with integrated reasoning

## Quality Assurance
- **Automated Verification**: All code solutions verified using execution environments and testing suites
- **Correctness Guarantee**: Only problems with verified correct and functional code are included
- **Human Review**: Sample validation by coding experts
- **Automated Checks**: Static analysis, linting, and runtime verification where applicable
- **Open and Closed-Source Filtering**: Integration of proprietary and community-driven tools for enhanced quality control

## Performance Metrics
Models trained on this dataset show significant improvements in:
- Code generation accuracy with reasoning
- Efficiency in producing detailed, step-by-step solutions
- Problem-solving speed and logical coherence
- Cross-language and cross-domain code transfer
- Reduction in hallucinated or erroneous code outputs through better reasoning

## Acknowledgments
Special thanks to our partners and contributors:
- **NVIDIA** - Reference datasets; CodeX contains many examples taken from NVIDIA's existing datasets
- **XenArcAI Team** - Dataset curation, quality assurance, along with customly generated examples

## Citation
**Anyone** can freely use and modify this dataset.

## License
This dataset is released under [apache-2.0].

```bibtex
@dataset{codex2024,
title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},
author={Parvesh at XenArcAI},
year={2024},
publisher={XenArcAI},
url={https://huggingface.co/datasets/XenArcAI/CodeX-2M-Thinking}
}
```

## Contact
For questions, suggestions, or collaboration opportunities:
- **Email**: [XenArcAI]([e-mail])
- **Twitter**: [@XenArcAI]
- **GitHub**: [XenArcAI]

---
*Built with ❤️ by XenArcAI - Advancing AI through high-quality data*

许可证：Apache-2.0
展示名称：CodeX-5M-Thinking
数据集名称：XenArcAI/CodeX-5M-Thinking
规模分类：100万<样本量<1000万
语言：英语
任务类别：
- 文本生成
- 问答
标签：
- 代码（Coding）
- 代码（Code）
- CodeX
- XenArcAI
- 大语言模型训练（LLM-training）
- 合成数据（synthetic）
- 精选数据集（curated）
- 基准测试集（benchmark）
- 推理数据集（reasoning-dataset）
- 数据工件（artifact）
注释创建者：
- 机器生成
- 专家验证
源数据集：
- XenArcAI内部合成生成
多语言属性：
- 单语言
---
# XenArcAI

<p align="center">
<img
src="https://cdn-uploads.huggingface.co/production/uploads/677fcdf29b9a9863eba3f29f/ZP4YDDIRewH5M-jKmE4Rt.png"
alt="CodeX 横幅"
width="70%"
style="border-radius:15px;"
/>

> 注：本数据集属于XenArcAI推出的CodeX系列数据集。该系列下包含多款数据集，核心目标是为模型训练与微调提供高质量的数据集资源。

本数据集为全合成数据集，先从优质公开数据源中精选样本，再结合闭源与开源大语言模型生成的合成数据进行增强。其可作为指令微调与模型微调的坚实基础，是目前针对带推理的编码任务最精炼、最全面的语料库之一。

### 核心特性
- **规模**：包含200万条经过严格精选的编码数据样本
- **多样性**：覆盖从基础语法到高级软件工程的全编程领域
- **质量**：采用多阶段过滤与验证流程，包含基于排序的过滤及专家遴选机制
- **推理聚焦**：回复中包含逐步推理过程，针对带有详细思考流程的指令训练进行了优化
- **准确性**：通过自动化测试框架验证代码执行与正确性

## 数据集概览
**CodeX-2M-Thinking** 是一款经过精心精选（curated）的编码数据集，专为基于指令的模型微调以及提升现有模型的代码生成与推理能力而设计。本全合成数据集是Hugging Face平台上规模庞大且经过全面过滤的编码语料库，强调采用逐步推理的思考模式以支撑更深层次的模型训练。

## 使用方法
bash
pip install -U datasets fsspec

python
from datasets import load_dataset
dataset = load_dataset("XenArcAI/CodeX-2M-Thinking")

### 核心特性
- **规模**：包含200万条经过严格精选的编码数据样本
- **多样性**：覆盖从基础语法到高级软件工程的全编程领域
- **质量**：采用多阶段过滤与验证流程，包含基于排序的过滤及专家遴选机制
- **推理聚焦**：回复中包含逐步推理过程，针对带有详细思考流程的指令训练进行了优化
- **准确性**：通过自动化测试框架验证代码执行与正确性

## 数据精选流程
本数据集通过全合成方式精心构建，通过选择性生成与合并样本，为生成式模型丰富数据集内容。

### 数据来源
- **优质现有数据集**：从线上多款优质编码数据集精选而来（例如NVIDIA的数据集以及XenArcAI内部数据集合集）
- **合成生成**：通过闭源与开源大语言模型（XenArcAI）全量生成
- **专家验证**：由XenArcAI团队的人工审核代码解决方案、推理过程与实现细节

### 过滤流水线
我们采用严格的过滤流程，整合闭源与开源过滤技术，确保仅保留最高质量的样本：
1. **去重**：移除重复的问题与代码解决方案
2. **标准化**：统一代码格式并清理语法错误
3. **停用词处理**：智能移除非必要注释与样板代码
4. **质量评分**：基于代码复杂度、可读性与效率等指标进行多维度质量评估
5. **基于排序的过滤**：采用高级排序算法，根据相关性、新颖性与实用性对优质样本进行优先级排序
6. **专家遴选**：由编码领域专家手动精选典型样本
7. **答案验证**：通过pytest等自动化测试框架进行测试与执行验证
8. **内容过滤**：移除不当、过时或存在错误的代码
9. **多样性平衡**：通过算法采样确保各编程语言与领域的样本分布均衡

### 问题复杂度分布
- **基础级别（30%）**：涵盖基础编程概念、简单语法与基础操作
- **中级级别（30%）**：需要模块化代码与基础算法的多函数问题
- **高级级别（40%）**：涉及数据结构、优化与系统设计的复杂挑战

### 覆盖的编程领域
- 算法与数据结构
- Web开发与框架
- 机器学习与人工智能实现
- 系统编程与操作系统
- 数据库管理与SQL/NoSQL
- 软件工程最佳实践
- 竞赛编程题目
> 注：上述领域仅作参考。实际数据集多样性极强，覆盖范围远超上述列举内容，且包含比描述更复杂的高阶问题，支持Python、Java、C++、JavaScript等多种编程语言。

## 应用场景
- **微调**：微调大语言模型的代码生成与推理能力
- **训练**：训练以编码与推理为核心的指令遵循模型
- **基准测试**：针对编码任务、问题解决与逻辑推理场景测试模型性能
- **研究**：用于AI辅助编程、自动代码补全与可解释AI等领域的研究
- **教育应用**：适用于需要逐步代码解释与推理过程的教育场景

## 数据集格式
每条样本包含以下内容：
- **问题描述**：清晰的编码挑战或任务说明
- **逐步解决方案**：详细的推理过程
- **代码解决方案**：集成推理过程的可执行最终代码

## 质量保障
- **自动化验证**：所有代码解决方案均通过执行环境与测试套件进行验证
- **正确性保障**：仅收录经过验证的正确可用代码对应的问题
- **人工审核**：由编码领域专家对样本进行验证
- **自动化检查**：针对适用场景进行静态分析、代码检查与运行时验证
- **闭源与开源过滤结合**：整合专有工具与社区驱动工具以强化质量管控

## 性能指标
在本数据集上训练的模型在以下方面表现出显著提升：
- 带推理的代码生成准确性
- 生成详细逐步解决方案的效率
- 问题解决速度与逻辑连贯性
- 跨语言与跨领域的代码迁移能力
- 通过更优的推理减少幻觉与错误代码输出

## 致谢
特别感谢以下合作伙伴与贡献者：
- **NVIDIA**：提供参考数据集；CodeX系列包含大量取自NVIDIA现有数据集的样本
- **XenArcAI团队**：负责数据集精选、质量保障以及定制化样本生成

## 引用
任何个人或机构均可自由使用与修改本数据集。

## 许可证
本数据集采用 [Apache-2.0] 许可证发布。

bibtex
@dataset{codex2024,
title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},
author={Parvesh at XenArcAI},
year={2024},
publisher={XenArcAI},
url={https://huggingface.co/datasets/XenArcAI/CodeX-2M-Thinking}
}

## 联系方式
如有疑问、建议或合作意向，请联系：
- **邮箱**：[XenArcAI]([e-mail])
- **Twitter**：[@XenArcAI]
- **GitHub**：[XenArcAI]

---
*由XenArcAI倾心打造——以高质量数据推动AI发展*