Ik45/data-science-en-id

---
license: mit
task_categories:
- text-generation
language:
- en
- id
base_model:
- wikimedia/wikipedia
- ccdv/arxiv-summarization
tags:
- data-science
- scientific
- machine-translation
- nlp
- wikipedia-subset
- arxiv-subset
dataset_info:
splits:
- name: train
size_categories:
- 10M<n<100M
---

# Data Science EN-ID Parallel Corpus (Scientific Domain)

## Dataset Description
This dataset is a curated English-Indonesian (EN-ID) parallel corpus specifically designed for the **Scientific** and **Data Science** domains. It was developed to support the training of Machine Translation (NMT) models and Large Language Models (LLMs) to better handle technical terminology, academic structures, and formal scientific language.

- **Primary Languages:** English (EN) and Indonesian (ID)
- **Domain:** Data Science, Artificial Intelligence, Machine Learning, and General Science.
- **Applications:** Neural Machine Translation, Domain Adaptation, Cross-lingual Information Retrieval.

## Source Data & Origin
This dataset is a specialized extension and refined subset derived from two primary high-quality sources available on Hugging Face:

1. **[wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia):** We extracted the English-Indonesian parallel subsets, specifically focusing on articles categorized under Science, Technology, and Mathematics.
2. **[ccdv/arxiv-summarization](https://huggingface.co/datasets/ccdv/arxiv-summarization):** We utilized the Arxiv metadata and document summaries to build a robust scientific corpus, filtering for Computer Science and Data Science domains.

## Extraction & Refinement Pipeline
To transform these general-purpose datasets into a domain-specific parallel corpus, the following pipeline was implemented:

### 1. Domain-Specific Filtering
Instead of using the entire Wikipedia or Arxiv dump, we applied a **Lexical Filter** using a dictionary of 500+ technical keywords. A sentence pair is only included if it contains terminology relevant to Data Science (e.g., *Neural Networks, Statistical Inference, Heuristics*).

### 2. Scientific Alignment
For the Arxiv data, we performed a custom alignment process to pair English scientific abstracts with their Indonesian technical equivalents, ensuring that the formal tone and academic nomenclature are preserved.

### 3. Noise Reduction (Regex-Based)
Since Arxiv and Wikipedia data often contain LaTeX code, citations (e.g., `[1]`, `(Author, 2023)`), and HTML artifacts, we applied a rigorous cleaning script to ensure the final output consists of clean, natural language sentences.

### 4. Deduplication
We cross-referenced both sources to remove overlapping entries, ensuring that the `Ik45/data-science-en-id` corpus is diverse and free from redundant training signals.

## Creation Methodology
Based on the processing pipeline in `ScriptTextTranslationDomainIlmiah.ipynb`, the dataset was constructed through several key stages:

### 1. Data Collection
The corpus was aggregated from various scientific articles , academic papers, and technical datasets. The focus was strictly maintained on high-quality technical content to ensure domain relevance.

### 2. Preprocessing & Data Cleaning
- **Deduplication:** Removal of redundant pairs to ensure data diversity and prevent model overfitting.
- **Noise Reduction:** Cleaning of non-textual characters, artifacts from PDF extractions, and broken symbols.
- **Normalization:** Standardizing text formats to maintain consistency across the entire corpus.

### 3. Alignment
Rigorous sentence-level alignment was performed to ensure that English technical terms are correctly mapped to their appropriate Indonesian counterparts within a scientific context.

## Usage
You can easily load this dataset using the Hugging Face `datasets` library:

```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("Ik45/data-science-en-id")

# Preview a sample
print(dataset['train'][0])
```

---
许可证：MIT许可证
任务类别：文本生成
语言：英语、印尼语
基础模型：wikimedia/wikipedia、ccdv/arxiv-summarization
标签：数据科学、科研、机器翻译、自然语言处理、维基百科子集、arXiv子集
数据集信息：
拆分：
- 名称：训练集
规模类别：1000万<样本数<1亿
---

# 数据科学英-印尼语平行语料库（科研领域）

## 数据集说明
本数据集是一款精心甄选的英语-印尼语（EN-ID）平行语料库，专为科研与数据科学领域打造，旨在支撑机器翻译（Machine Translation）模型与大语言模型（Large Language Model, LLM）的训练，使其能够更好地处理专业技术术语、学术结构与正式科研语体。
- **主要语言：** 英语（EN）与印尼语（ID）
- **覆盖领域：** 数据科学、人工智能、机器学习与通用科学
- **应用场景：** 神经机器翻译（Neural Machine Translation, NMT）、领域自适应、跨语言信息检索

## 源数据与来源
本数据集是基于Hugging Face平台上两款优质源数据集进行专业化扩展与精细化子集筛选后得到的：
1. **[wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)：** 我们提取了其中的英语-印尼语平行子集，重点聚焦于科学、技术与数学分类下的词条。
2. **[ccdv/arxiv-summarization](https://huggingface.co/datasets/ccdv/arxiv-summarization)：** 我们利用该数据集的arXiv元数据与文档摘要构建了高质量的科研语料库，并筛选出计算机科学与数据科学领域的内容。

## 提取与优化流程
为将这些通用型数据集转换为领域专属平行语料库，我们实施了以下处理流程：
### 1. 领域专属筛选
我们并未直接使用完整的维基百科或arXiv数据集，而是借助包含500+专业关键词的词典实现了**词汇筛选器**。仅当句子对包含数据科学相关术语（如神经网络、统计推断、启发式算法）时，才会被纳入语料库。
### 2. 科研语料对齐
针对arXiv数据，我们采用自定义对齐流程，将英语科研摘要与其对应的印尼语技术译文进行配对，确保保留正式语体与学术命名规范。
### 3. 噪声去除（基于正则表达式）
由于维基百科与arXiv数据通常包含LaTeX代码、引用格式（如`[1]`、`(作者, 2023)`）与HTML残留元素，我们编写了严格的清理脚本，确保最终输出均为干净自然的语言文本。
### 4. 去重
我们对两个源数据集进行交叉比对，移除重复条目，确保`Ik45/data-science-en-id`语料库具备多样性且无冗余训练信号。

## 构建方法
基于`ScriptTextTranslationDomainIlmiah.ipynb`中的处理流程，本数据集通过以下关键阶段构建完成：
### 1. 数据收集
本语料库聚合自各类科研文章、学术论文与技术数据集，全程严格聚焦高质量技术内容，以确保领域相关性。
### 2. 预处理与数据清洗
- **去重：** 移除冗余样本对，保障数据多样性并防止模型过拟合。
- **噪声去除：** 清理非文本字符、PDF提取残留的伪影与损坏符号。
- **规范化：** 统一文本格式，确保整个语料库的格式一致性。
### 3. 语料对齐
我们实施了严格的句子级对齐流程，确保英语技术术语在科研语境下能够正确映射至对应的印尼语对等表达。

## 使用方法
您可通过Hugging Face的`datasets`库轻松加载本数据集：
python
from datasets import load_dataset

# 加载数据集
dataset = load_dataset("Ik45/data-science-en-id")

# 预览样本
print(dataset['train'][0])