MnemicAI/Ling-Coder-SFT-English-Clean

---
license: apache-2.0
task_categories:
- text-generation
language:
- en
tags:
- code
- coding
- sft
- instruction-tuning
- english-only
- cleaned
- curated
pretty_name: "Ling-Coder SFT English Clean"
size_categories:
- 1M<n<5M
source_datasets:
- inclusionAI/Ling-Coder-SFT
dataset_info:
features:
- name: messages
dtype: string
- name: languages
dtype: string
- name: license
dtype: string
- name: difficulty
dtype: string
configs:
- config_name: default
data_files:
- split: train
path: "**/*.parquet"
---

# Ling-Coder-SFT-English-Clean

A cleaned, English-only version of [inclusionAI/Ling-Coder-SFT](https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT) — one of the largest open-source coding instruction datasets (~5.1M samples). Split by programming language for easy access.

**Curated by [MnemicAI](https://huggingface.co/MnemicAI)**

## Origin Story

While building our [Mnemic COCM-COT](https://huggingface.co/MnemicAI) training pipeline — a multi-language coding instruction dataset with stratified topic sampling — we discovered that **11.44% of Ling-Coder-SFT contains Chinese/CJK characters** mixed into what many assume is an English-only coding dataset.

This wasn't a bug in the original dataset. InclusionAI intentionally built it as bilingual (English + Chinese) for their Ling-Coder-Lite model. But for anyone fine-tuning an **English-only** code model, these 585,600 samples silently contaminate your training data and can degrade model quality.

Since we were already scanning 5.1M rows for our own pipeline, we figured — why not clean the whole thing and share it? It took 25 minutes on a free Colab instance.

**If this saves you time, give it a ⭐ and build something great.**

## What Changed

We scanned all 5,119,470 rows and removed every row containing Chinese/CJK characters.

| Metric | Original | This Version |
|--------|----------|--------------|
| Total rows | 5,119,470 | **4,533,870** |
| Chinese removed | 585,600 (11.44%) | **0** |
| Programming languages | 293 | 293 (unchanged) |
| Schema | Unchanged | Unchanged |
| License | Apache 2.0 | Apache 2.0 |

### Per-Language Breakdown (Top 25)

| Language | Rows | % of Dataset |
|----------|-----:|:-------------|
| Python | 3,156,935 | 69.6% |
| JavaScript | 116,446 | 2.6% |
| C# | 109,671 | 2.4% |
| C++ | 87,574 | 1.9% |
| Java | 82,112 | 1.8% |
| Go | 73,501 | 1.6% |
| Swift | 68,301 | 1.5% |
| Rust | 64,011 | 1.4% |
| TypeScript | 62,461 | 1.4% |
| PHP | 56,389 | 1.2% |
| D | 54,281 | 1.2% |
| R | 52,865 | 1.2% |
| Clojure | 51,098 | 1.1% |
| Bash | 51,018 | 1.1% |
| Lua | 45,281 | 1.0% |
| Haskell | 44,311 | 1.0% |
| Elixir | 43,639 | 1.0% |
| Scala | 41,572 | 0.9% |
| Julia | 41,408 | 0.9% |
| SQL | 40,939 | 0.9% |
| Ruby | 36,081 | 0.8% |
| Racket | 31,952 | 0.7% |
| C | 27,128 | 0.6% |
| Kotlin | 26,776 | 0.6% |
| HTML | 21,648 | 0.5% |

*...and 268 more languages including Dart, Solidity, Perl, Dockerfile, YAML, and others.*

## Dataset Structure

The dataset is organized by **programming language**, making it easy to download only what you need:

```
├── python/
│ ├── train-00000-of-00003.parquet
│ ├── train-00001-of-00003.parquet
│ └── train-00002-of-00003.parquet
├── java/
│ └── train-00000-of-00001.parquet
├── typescript/
│ └── train-00000-of-00001.parquet
├── rust/
│ └── train-00000-of-00001.parquet
├── go/
│ └── train-00000-of-00001.parquet
├── cpp/
│ └── train-00000-of-00001.parquet
├── csharp/
│ └── train-00000-of-00001.parquet
├── swift/
│ └── train-00000-of-00001.parquet
├── kotlin/
│ └── train-00000-of-00001.parquet
└── ... (20+ languages)
```

## Usage

### Load the full dataset
```python
from datasets import load_dataset

ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean")
```

### Load a specific language only
```python
from datasets import load_dataset

# Load only Python samples
python_ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean", data_dir="python")

# Load only Rust samples
rust_ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean", data_dir="rust")
```

### Example Row
```json
{
"messages": [
{"role": "user", "content": "Write a Python function to find the longest common subsequence..."},
{"role": "assistant", "content": "Here's a dynamic programming solution..."}
],
"languages": ["python"],
"license": "MIT",
"difficulty": "medium"
}
```

## Filtering Methodology

Our filtering pipeline uses a **zero-regex, C-level** approach for maximum speed:

```python
# Pre-built frozenset of 20,992 CJK codepoints (Unicode range U+4E00–U+9FFF)
_CJK_CHARS = frozenset(chr(c) for c in range(0x4e00, 0x9fff + 1))

def has_chinese(text):
"""C-level set intersection — no Python loops, no regex."""
return not _CJK_CHARS.isdisjoint(str(text))
```

- **Every message** in each row is scanned (user + assistant turns)
- If **any** message contains CJK characters, the entire row is removed
- Processing speed: ~6,000 rows/second on a free Colab instance
- Total processing time: ~15 minutes for 5.1M rows

### What Gets Filtered

- ❌ Instructions written entirely in Chinese
- ❌ Responses containing Chinese explanations mixed with code
- ❌ Mixed English-Chinese bilingual samples
- ✅ Code comments in English (kept)
- ✅ All programming language syntax (kept)
- ✅ Unicode strings in code examples that aren't CJK (kept)

## Intended Use

This dataset is designed for:

- 🎯 **Fine-tuning English-only code models** (SFT stage)
- 🎯 **Building coding assistants** that don't need Chinese support
- 🎯 **Research** on code generation and instruction following
- 🎯 **Language-specific training** (grab just the language folder you need)

## Attribution & License

This dataset is a filtered derivative of [inclusionAI/Ling-Coder-SFT](https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT), licensed under **Apache License 2.0**.

**Changes made:**
- Removed 585,600 rows containing Chinese/CJK characters (11.44% of the original dataset)
- Split into per-programming-language subdirectories
- No other modifications to the data

**Original dataset citation:**
```bibtex
@misc{lingcoder2025,
title={Ling-Coder: An Instruction-Tuned Code Large Language Model},
author={InclusionAI},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT}
}
```

**Curated by:** [MnemicAI](https://huggingface.co/MnemicAI)

---

许可证：Apache-2.0
任务类别：
- 文本生成
语言：
- 英语
标签：
- 代码、编码、监督微调（Supervised Fine-Tuning，SFT）、指令微调（instruction-tuning）、仅英语、已清洗、已精选
美观名称："Ling-Coder SFT 英文清洗版"
样本规模：100万 < 样本数 < 500万
源数据集：
- inclusionAI/Ling-Coder-SFT
数据集信息：
字段：
- 名称：messages，数据类型：字符串
- 名称：编程语言（languages），数据类型：字符串
- 名称：许可证（license），数据类型：字符串
- 名称：难度（difficulty），数据类型：字符串
配置：
- 配置名称：默认（default）
数据文件：
- 拆分：训练集（train）
路径："**/*.parquet"

# Ling-Coder-SFT 英文清洗版

本数据集是[inclusionAI/Ling-Coder-SFT](https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT)的已清洗仅英语版本，该源数据集是目前规模最大的开源代码指令数据集之一（约510万条样本）。本数据集按编程语言拆分，便于按需获取。

**由[MnemicAI](https://huggingface.co/MnemicAI)精选整理**

## 创作背景
在搭建我们的[Mnemic COCM-COT](https://huggingface.co/MnemicAI)训练流水线（该流水线是采用分层主题采样的多语言代码指令数据集）时，我们发现**Ling-Coder-SFT数据集中有11.44%的样本包含中文字符/中日韩统一表意文字（CJK）**，而许多人原本认为这是一个纯英语的代码数据集。

这并非源数据集的漏洞：InclusionAI原本是为其Ling-Coder-Lite模型构建的双语（英语+中文）数据集。但对于想要微调**纯英语**代码模型的开发者而言，这585600条样本会在不知不觉中污染训练数据，进而降低模型性能。

既然我们已经在为自有流水线扫描510万条样本，我们便思考：何不将整个数据集清洗后分享出来？在免费的Colab实例上仅耗时25分钟即可完成清洗。

**若本数据集为您节省了时间，请点亮⭐并打造出色的成果。**

## 变更内容
我们扫描了全部5119470条样本，并移除了所有包含中文字符/中日韩统一表意文字的样本。

| 指标 | 原始数据集 | 本版本 |
| ---- | -------- | ------ |
| 总样本数 | 5,119,470 | **4,533,870** |
| 移除中文字符样本数 | 585,600（占比11.44%） | **0** |
| 支持编程语言数量 | 293 | 293（无变化） |
| 数据结构 | 未变更 | 未变更 |
| 许可证 | Apache 2.0 | Apache 2.0 |

### 按编程语言分布（前25名）
| 编程语言 | 样本数 | 占数据集比例 |
|----------|-----:|:-------------|
| Python | 3,156,935 | 69.6% |
| JavaScript | 116,446 | 2.6% |
| C# | 109,671 | 2.4% |
| C++ | 87,574 | 1.9% |
| Java | 82,112 | 1.8% |
| Go | 73,501 | 1.6% |
| Swift | 68,301 | 1.5% |
| Rust | 64,011 | 1.4% |
| TypeScript | 62,461 | 1.4% |
| PHP | 56,389 | 1.2% |
| D | 54,281 | 1.2% |
| R | 52,865 | 1.2% |
| Clojure | 51,098 | 1.1% |
| Bash | 51,018 | 1.1% |
| Lua | 45,281 | 1.0% |
| Haskell | 44,311 | 1.0% |
| Elixir | 43,639 | 1.0% |
| Scala | 41,572 | 0.9% |
| Julia | 41,408 | 0.9% |
| SQL | 40,939 | 0.9% |
| Ruby | 36,081 | 0.8% |
| Racket | 31,952 | 0.7% |
| C | 27,128 | 0.6% |
| Kotlin | 26,776 | 0.6% |
| HTML | 21,648 | 0.5% |

*……另有268种编程语言，包括Dart、Solidity、Perl、Dockerfile、YAML等。*

## 数据集结构
本数据集按**编程语言**进行组织，便于用户按需下载所需数据：

├── python/
│ ├── train-00000-of-00003.parquet
│ ├── train-00001-of-00003.parquet
│ └── train-00002-of-00003.parquet
├── java/
│ └── train-00000-of-00001.parquet
├── typescript/
│ └── train-00000-of-00001.parquet
├── rust/
│ └── train-00000-of-00001.parquet
├── go/
│ └── train-00000-of-00001.parquet
├── cpp/
│ └── train-00000-of-00001.parquet
├── csharp/
│ └── train-00000-of-00001.parquet
├── swift/
│ └── train-00000-of-00001.parquet
├── kotlin/
│ └── train-00000-of-00001.parquet
└── ... (20+ languages)

## 使用方法
### 加载完整数据集
python
from datasets import load_dataset

ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean")

### 仅加载特定编程语言的样本
python
from datasets import load_dataset

# 仅加载Python样本
python_ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean", data_dir="python")

# 仅加载Rust样本
rust_ds = load_dataset("MnemicAI/Ling-Coder-SFT-English-Clean", data_dir="rust")

### 样本示例
json
{
"messages": [
{"role": "user", "content": "Write a Python function to find the longest common subsequence..."},
{"role": "assistant", "content": "Here's a dynamic programming solution..."}
],
"languages": ["python"],
"license": "MIT",
"difficulty": "medium"
}

## 过滤方法
我们的过滤流水线采用**零正则表达式、C语言级**的实现方式以获得最高速度：
python
# 预构建包含20992个中日韩统一表意文字码位的不可变集合（Unicode范围U+4E00–U+9FFF）
_CJK_CHARS = frozenset(chr(c) for c in range(0x4e00, 0x9fff + 1))

def has_chinese(text):
"""C语言级集合交集操作——无Python循环、无正则表达式。"""
return not _CJK_CHARS.isdisjoint(str(text))

- 扫描每条样本中的**所有消息**（包括用户提问与助手回复）
- 若**任意一条**消息包含中日韩统一表意文字，则移除整条样本
- 处理速度：在免费Colab实例上约为6000条/秒
- 总处理时间：处理510万条样本仅需约15分钟

### 被过滤的样本类型
- ❌ 纯中文的指令
- ❌ 混杂中文解释与代码的回复
- ❌ 中英混合的双语样本
- ✅ 英语注释的代码（保留）
- ✅ 所有编程语言语法（保留）
- ✅ 代码示例中非中日韩统一表意文字的Unicode字符串（保留）

## 预期用途
本数据集适用于：
- 🎯 **微调纯英语代码模型**（监督微调（SFT）阶段）
- 🎯 **构建无需中文支持的代码助手**
- 🎯 **代码生成与指令遵循相关研究**
- 🎯 **特定编程语言训练**（按需下载对应语言的文件夹）

## 署名与许可证
本数据集是[inclusionAI/Ling-Coder-SFT](https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT)的过滤衍生版本，采用**Apache许可证2.0**授权。

**所做变更：**
- 移除了585600条包含中日韩统一表意文字的样本（占原始数据集的11.44%）
- 按编程语言拆分为子目录
- 未对数据进行其他修改

**源数据集引用：**
bibtex
@misc{lingcoder2025,
title={Ling-Coder: An Instruction-Tuned Code Large Language Model},
author={InclusionAI},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/datasets/inclusionAI/Ling-Coder-SFT}
}

**精选整理：** [MnemicAI](https://huggingface.co/MnemicAI)