XenArcAI/CodeX-2M-Thinking

---
license: apache-2.0
pretty_name: CodeX-5M-Thinking
dataset_name: XenArcAI/CodeX-5M-Thinking
size_categories:
- 1M<n<10M
language:
- en
task_categories:
- text-generation
- question-answering
tags:
- Coding
- Code
- CodeX
- XenArcAI
- LLM-training
- synthetic
- curated
- benchmark
- reasoning-dataset
- artifact
annotations_creators:
- machine-generated
- expert-verified
source_datasets:
- XenArcAI internal synthetic generation
multilinguality:
- monolingual
---
# XenArcAI
---

<p align="center">
<img
src="https://cdn-uploads.huggingface.co/production/uploads/677fcdf29b9a9863eba3f29f/ZP4YDDIRewH5M-jKmE4Rt.png"
alt="CodeX Banner"
width="70%"
style="border-radius:15px;"
/>

> Note: This dataset is part of the lineup CodeX by XenArcAI. You can get lots of datasets in this same lineup, with the main focus on providing very high-quality datasets for model training and fine-tuning.

This dataset is fully synthetic, curated from high-quality public sources and enhanced with synthetic data generated using both closed and open-source models. It serves as a strong foundation for instruction-based model tuning and fine-tuning, offering one of the most refined and extensive corpora available for coding tasks with reasoning.

### Key Features
- **Scale**: 2 million examples of highly curated coding data
- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks

## Dataset Overview
**CodeX-2M-Thinking** is a meticulously curated coding dataset designed specifically for instruction-based model tuning and fine-tuning of existing models with enhanced code generation and reasoning capabilities. This fully synthetic dataset represents a large and comprehensively filtered corpus of coding data on the Hugging Face platform, emphasizing a thinking approach with step-by-step reasoning for deeper model training.

## How to Use?
```bash
pip install -U datasets fsspec
```
```python
from datasets import load_dataset
dataset = load_dataset("XenArcAI/CodeX-2M-Thinking")
```

### Key Features
- **Scale**: 2 million examples of highly curated coding data
- **Diversity**: Comprehensive coverage of programming domains from basic syntax to advanced software engineering
- **Quality**: Multi-stage filtering and verification processes, including ranking-based filtering and expert selections
- **Thinking Focus**: Step-by-step reasoning included in responses, optimized for instruction training with detailed thought processes
- **Accuracy**: Verified code executions and correctness validation using automated testing frameworks

## Data Curation Process
This dataset has been carefully constructed through a fully synthetic approach, selectively generating and merging examples to enrich the overall dataset for generation models.

### Data Sources
- **High-Quality Existing Datasets**: Curated from multiple premium coding datasets available online (e.g., from NVIDIA and XenArcAI's internal collections)
- **Synthetic Generation**: Fully generated using both closed-source and open-source language models (XenArcAI)
- **Expert Validation**: Human-verified code solutions, reasoning, and implementations (XenArcAI)

### Filtering Pipeline
Our rigorous filtering process includes open and closed-source filtering techniques, ensuring only the highest-quality examples are retained:
1. **Deduplication**: Removal of duplicate problems and code solutions
2. **Normalization**: Code formatting standardization and syntax cleanup
3. **Stopword Processing**: Intelligent removal of non-essential comments or boilerplate
4. **Quality Scoring**: Multi-dimensional quality assessment using metrics like code complexity, readability, and efficiency
5. **Ranking-Based Filtering**: Advanced ranking algorithms to prioritize top-tier examples based on relevance, novelty, and utility
6. **Expert Selections**: Manual curation by coding experts to select exemplary samples
7. **Answer Verification**: Automated testing and execution validation using frameworks like pytest or unit tests
8. **Content Filtering**: Removal of inappropriate, outdated, or incorrect code
9. **Diversity Balancing**: Ensuring balanced representation across languages and domains through algorithmic sampling

### Problem Complexity Distribution
- **Basic Level** (30%): Fundamental programming concepts, simple syntax, and basic operations
- **Intermediate Level** (30%): Multi-function problems requiring modular code and basic algorithms
- **Advanced Level** (40%): Complex challenges involving data structures, optimization, and system design

### Programming Domains Covered
- Algorithms and Data Structures
- Web Development and Frameworks
- Machine Learning and AI Implementations
- System Programming and Operating Systems
- Database Management and SQL/NoSQL
- Software Engineering Best Practices
- Competitive Programming Problems
> Note: Domains are for reference only. The actual data is very diverse and covers more domains than stated. The actual data includes more complex and high-level questions than stated, spanning multiple programming languages such as Python, Java, C++, JavaScript, and others.

## Use Cases
- **Fine-tuning** code generation and reasoning capabilities in language models
- **Training** instruction-following models with a coding and reasoning focus
- **Benchmarking** model performance on coding tasks, problem-solving, and logical reasoning
- **Research** in AI-assisted programming, automated code completion, and explainable AI
- **Educational** applications requiring step-by-step code explanations and reasoning

## Dataset Format
Each example contains:
- **Problem Statement**: Clear coding challenge or task description
- **Step-by-Step Solution**: Detailed reasoning process
- **Code Solution**: Final executable code with integrated reasoning

## Quality Assurance
- **Automated Verification**: All code solutions verified using execution environments and testing suites
- **Correctness Guarantee**: Only problems with verified correct and functional code are included
- **Human Review**: Sample validation by coding experts
- **Automated Checks**: Static analysis, linting, and runtime verification where applicable
- **Open and Closed-Source Filtering**: Integration of proprietary and community-driven tools for enhanced quality control

## Performance Metrics
Models trained on this dataset show significant improvements in:
- Code generation accuracy with reasoning
- Efficiency in producing detailed, step-by-step solutions
- Problem-solving speed and logical coherence
- Cross-language and cross-domain code transfer
- Reduction in hallucinated or erroneous code outputs through better reasoning

## Acknowledgments
Special thanks to our partners and contributors:
- **NVIDIA** - Reference datasets; CodeX contains many examples taken from NVIDIA's existing datasets
- **XenArcAI Team** - Dataset curation, quality assurance, along with customly generated examples

## Citation
**Anyone** can freely use and modify this dataset.

## License
This dataset is released under [apache-2.0].

```bibtex
@dataset{codex2024,
title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},
author={Parvesh at XenArcAI},
year={2024},
publisher={XenArcAI},
url={https://huggingface.co/datasets/XenArcAI/CodeX-2M-Thinking}
}
```

## Contact
For questions, suggestions, or collaboration opportunities:
- **Email**: [XenArcAI]([e-mail])
- **Twitter**: [@XenArcAI]
- **GitHub**: [XenArcAI]

---
*Built with ❤️ by XenArcAI - Advancing AI through high-quality data*

license: apache-2.0
pretty_name: CodeX-5M-Thinking
dataset_name: XenArcAI/CodeX-5M-Thinking
size_categories:
- 100万<样本数<1000万
language:
- 英语
task_categories:
- 文本生成
- 问答
tags:
- 编程（Coding）
- 代码（Code）
- CodeX
- XenArcAI
- 大语言模型（LLM）训练
- 合成数据集
- 精选数据集
- 基准测试
- 推理数据集（reasoning-dataset）
- 工件
annotations_creators:
- 机器生成
- 专家验证
source_datasets:
- XenArcAI内部合成生成
multilinguality:
- 单语言

# XenArcAI

<p align="center">
<img
src="https://cdn-uploads.huggingface.co/production/uploads/677fcdf29b9a9863eba3f29f/ZP4YDDIRewH5M-jKmE4Rt.png"
alt="CodeX 横幅"
width="70%"
style="border-radius:15px;"
/>

> 注：本数据集属于XenArcAI推出的CodeX系列数据集。该系列包含多款优质数据集，核心目标是为模型训练与微调提供高质量数据支撑。

本数据集为全合成数据集，从优质公开数据源精选而来，并结合闭源与开源大语言模型（LLM）生成的合成数据进行增强。其可作为基于指令的模型微调的坚实基础，是当前支持推理的编码任务领域中最精细、最全面的语料库之一。

### 核心特性
- **规模**：包含200万条经过严格精选的编码数据样本
- **多样性**：覆盖从基础语法到高级软件工程的全编程领域
- **质量**：采用多阶段过滤与验证流程，包含基于排序的筛选与人工专家遴选
- **推理聚焦**：响应内容包含逐步推理过程，针对带有详细思维过程的指令训练进行了优化
- **准确性**：通过自动化测试框架验证代码执行与正确性

## 数据集概览
**CodeX-2M-Thinking** 是一款经过精心打磨的精选编码数据集，专为基于指令的模型微调以及提升现有模型的代码生成与推理能力而设计。本全合成数据集是Hugging Face平台上规模庞大、经过全面过滤的编码语料库，强调带有逐步推理的思维模式，以支持更深入的模型训练。

## 使用方法
bash
pip install -U datasets fsspec

python
from datasets import load_dataset
dataset = load_dataset("XenArcAI/CodeX-2M-Thinking")

### 核心特性
- **规模**：包含200万条经过严格精选的编码数据样本
- **多样性**：覆盖从基础语法到高级软件工程的全编程领域
- **质量**：采用多阶段过滤与验证流程，包含基于排序的筛选与人工专家遴选
- **推理聚焦**：响应内容包含逐步推理过程，针对带有详细思维过程的指令训练进行了优化
- **准确性**：通过自动化测试框架验证代码执行与正确性

## 数据精选流程
本数据集通过全合成路径精心构建，通过选择性生成与合并样本，为生成模型丰富数据集内容。

### 数据来源
- **高质量现有数据集**：从线上多款优质编码数据集精选而来（例如NVIDIA公开数据集与XenArcAI内部数据集）
- **合成生成**：由XenArcAI团队通过闭源与开源大语言模型（LLM）完全生成
- **专家验证**：由XenArcAI团队对代码解决方案、推理过程与实现进行人工核验

### 过滤流水线
我们的严格过滤流程整合了开源与闭源过滤技术，确保仅保留最高质量的样本：
1. **去重**：移除重复的问题与代码解决方案
2. **标准化**：统一代码格式并清理语法错误
3. **冗余处理**：智能移除非必要注释与样板代码
4. **质量评分**：基于代码复杂度、可读性与效率等指标进行多维度质量评估
5. **基于排序的筛选**：通过高级排序算法，根据相关性、新颖性与实用性优先保留优质样本
6. **专家遴选**：由编码领域专家进行人工精选，保留标杆样本
7. **答案验证**：使用pytest等单元测试框架进行自动化测试与执行验证
8. **内容过滤**：移除不当、过时或存在错误的代码
9. **多样性平衡**：通过算法采样确保各编程语言与领域的样本分布均衡

### 问题难度分布
- **基础难度（30%）**：涵盖基础编程概念、简单语法与基础操作
- **中级难度（30%）**：需要模块化代码与基础算法的多函数问题
- **高级难度（40%）**：涉及数据结构、优化与系统设计的复杂挑战

### 覆盖的编程领域
- 算法与数据结构
- Web开发与框架
- 机器学习与AI实现
- 系统编程与操作系统
- 数据库管理与SQL/NoSQL
- 软件工程最佳实践
- 竞赛编程题
> 注：上述领域仅作参考。实际数据集涵盖的领域远多于所列内容，且包含更复杂的高阶问题，支持Python、Java、C++、JavaScript等多种编程语言。

## 应用场景
- **微调场景**：用于微调大语言模型的代码生成与推理能力
- **训练场景**：训练以编码与推理为核心的指令跟随模型
- **基准测试**：用于评测模型在编码任务、问题解决与逻辑推理中的性能
- **研究场景**：用于AI辅助编程、自动代码补全与可解释AI等领域的研究
- **教育场景**：适用于需要逐步代码解释与推理过程的教育应用

## 数据集格式
每条样本包含以下内容：
- **问题描述**：清晰的编码挑战或任务说明
- **逐步解决方案**：详细的推理过程
- **代码解决方案**：集成了推理过程的可执行最终代码

## 质量保障
- **自动化验证**：所有代码解决方案均通过执行环境与测试套件进行验证
- **正确性保障**：仅收录经过验证的正确可用代码的问题
- **人工审核**：由编码领域专家对样本进行核验
- **自动化检查**：在适用场景下进行静态分析、代码风格检查与运行时验证
- **开源与闭源过滤整合**：整合专有工具与社区驱动工具，强化质量管控

## 性能指标
使用本数据集训练的模型在以下方面表现出显著提升：
- 带推理的代码生成准确率
- 生成详细逐步解决方案的效率
- 问题解决速度与逻辑连贯性
- 跨语言与跨领域的代码迁移能力
- 通过更优推理减少幻觉与错误代码输出

## 致谢
特别感谢以下合作伙伴与贡献者：
- **NVIDIA**：提供参考数据集；CodeX系列包含大量取自NVIDIA现有数据集的样本
- **XenArcAI团队**：负责数据集精选、质量保障与定制化样本生成

## 引用声明
任何人均可自由使用与修改本数据集。

## 许可证
本数据集基于Apache-2.0协议发布。

bibtex
@dataset{codex2024,
title={CodeX-2M-Thinking: Large-Scale Coding Dataset with Reasoning},
author={Parvesh at XenArcAI},
year={2024},
publisher={XenArcAI},
url={https://huggingface.co/datasets/XenArcAI/CodeX-2M-Thinking}
}

## 联系方式
如有疑问、建议或合作意向，请联系：
- **邮箱**：[XenArcAI]([e-mail])
- **Twitter**：[@XenArcAI]
- **GitHub**：[XenArcAI]

---
*由XenArcAI用心打造——以高质量数据推动AI发展*