lihwak74/egyptian-dialogue

--- language: - ar - en license: cc-by-4.0 task_categories: - translation - text-generation tags: - egyptian-arabic - dialect - colloquial - ar_EG - translation - dialogue - subtitles - domain-classification pretty_name: Egyptian Arabic Dialogue Dataset size_categories: - 1K<n<10K --- # Egyptian Arabic Dialogue Dataset ## Dataset Description This dataset contains **4,322 parallel Egyptian Arabic-English dialogue pairs** with automatic domain classification. The data is extracted from TV series subtitles and features natural conversational Egyptian Arabic dialect (العامية المصرية). ### Languages - **Source**: Egyptian Arabic (ar_EG) - Colloquial dialect - **Target**: English (en) ## Dataset Summary Egyptian Arabic is one of the most widely spoken Arabic dialects, used by over 100 million speakers. This dataset provides: - Natural conversational dialogue - Colloquial expressions and idioms - Domain-classified content for specialized training - Episode context for narrative understanding ## Dataset Structure ### Data Format Each entry contains: ```json { "id": "ep01_line0001", "arabic": "خلاويص؟", "english": "Ready or not?", "episode": 1, "dialect": "egyptian", "language": "ar", "language_variant": "ar_EG", "genre": "dialogue", "domain": "general" } ``` ### Data Fields | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique identifier (format: epXX_lineYYYY) | | `arabic` | string | Egyptian Arabic text | | `english` | string | English translation | | `episode` | int | Episode number (for context) | | `dialect` | string | Dialect identifier (always "egyptian") | | `language` | string | ISO language code (always "ar") | | `language_variant` | string | Specific variant code (always "ar_EG") | | `genre` | string | Content genre (dialogue/narration) | | `domain` | string | Auto-detected content domain | ## Dataset Statistics ### Overview - **Total Entries**: 4,322 - **Episodes**: 6 - **Unique Domains**: 18 - **Unique Genres**: 2 - **Average Arabic Length**: 25.9 characters - **Average English Length**: 35.0 characters ### Domain Distribution | Domain | Count | Percentage | |--------|-------|------------| | general | 2,143 | 49.6% | | technology | 531 | 12.3% | | family | 368 | 8.5% | | horror | 281 | 6.5% | | medical | 233 | 5.4% | | romance | 136 | 3.1% | | weather | 115 | 2.7% | | food | 104 | 2.4% | | paranormal | 86 | 2.0% | | social | 55 | 1.3% | ### Episode Distribution | Episode | Entries | |---------|---------| | Episode 1 | 889 | | Episode 2 | 782 | | Episode 3 | 584 | | Episode 4 | 907 | | Episode 5 | 554 | | Episode 6 | 606 | ### Genre Distribution - **dialogue**: 4,301 (99.5%) - **narration**: 21 (0.5%) ## Domains Explained This dataset includes **automatic domain classification** using keyword-based detection: - **general** - Everyday conversation without specific domain - **family** - Family relationships, relatives, marriage - **horror** - Scary themes, ghosts, supernatural fear - **medical** - Healthcare, doctors, treatment - **technology** - Computers, phones, internet, apps - **romance** - Love, relationships, emotions - **paranormal** - Mysterious, unexplained phenomena - **weather** - Climate, meteorology, temperature - **food** - Cooking, restaurants, meals - **social** - Friends, gatherings, social life - **crime** - Police, investigation, law enforcement - **education** - Schools, universities, learning - **sports** - Games, matches, tournaments - **entertainment** - Movies, series, cinema - **legal** - Law, court, legal matters - **news** - Journalism, reports, media - **business** - Companies, economy, trading - **politics** - Government, elections, policy ## Use Cases ### ✅ Recommended Use Cases - **Egyptian Arabic Translation**: Train translation models specifically for Egyptian dialect - **Domain-Specific Models**: Train models for specific domains (medical, legal, etc.) - **Dialect Studies**: Research on Egyptian Arabic characteristics - **Conversational AI**: Build chatbots for Egyptian users - **Language Modeling**: Pre-train or fine-tune on Egyptian dialect - **Multi-Domain Learning**: Train models aware of content domains ### ⚠️ Limitations - **Domain Scope**: Limited to entertainment/dialogue domain content - **Register**: Conversational/informal language only - **Size**: 4,322 entries (relatively small for large-scale pre-training) - **Dialect Variation**: Egyptian Arabic has regional sub-dialects not captured - **Context**: Individual dialogue lines may lack broader narrative context ## Loading the Dataset ### Using Hugging Face Datasets ```python from datasets import load_dataset # Load the dataset dataset = load_dataset("fr3on/egyptian-dialogue") # Access the data print(dataset['train'][0]) # Filter by domain medical_data = dataset['train'].filter(lambda x: x['domain'] == 'medical') # Filter by episode episode_1 = dataset['train'].filter(lambda x: x['episode'] == 1) ``` ### Using Pandas ```python import pandas as pd # Load Parquet file directly df = pd.read_parquet("data/train-00000-of-00001.parquet") # Analyze domains print(df['domain'].value_counts()) # Filter and export medical_df = df[df['domain'] == 'medical'] ``` ## Training Examples ### Translation Model ```python from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainer # Load dataset dataset = load_dataset("fr3on/egyptian-dialogue") # Load model for Arabic-English translation model_name = "Helsinki-NLP/opus-mt-ar-en" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSeq2SeqLM.from_pretrained(model_name) # Tokenize def preprocess(examples): inputs = tokenizer(examples['arabic'], truncation=True, max_length=128) targets = tokenizer(examples['english'], truncation=True, max_length=128) inputs['labels'] = targets['input_ids'] return inputs tokenized = dataset.map(preprocess, batched=True) # Train trainer = Seq2SeqTrainer( model=model, train_dataset=tokenized['train'], eval_dataset=tokenized['test'] ) trainer.train() ``` ### Domain-Aware Training ```python from datasets import load_dataset dataset = load_dataset("fr3on/egyptian-dialogue") # Train separate models per domain for domain in ['medical', 'legal', 'technology']: domain_data = dataset['train'].filter(lambda x: x['domain'] == domain) # Train domain-specific model print(f"Training {domain} model with {len(domain_data)} examples") ``` ## Data Collection & Processing ### Source - **Origin**: Egyptian TV series subtitles - **Language**: Professional subtitle translations - **Quality**: Natural, conversational Egyptian Arabic ### Processing Pipeline 1. **Extraction**: Load from Excel subtitle files 2. **Cleaning**: Remove empty rows, very short entries 3. **Deduplication**: Hash-based duplicate removal (945 duplicates removed) 4. **Domain Detection**: Automatic classification using keyword matching 5. **Genre Classification**: Automatic dialogue vs. narration detection 6. **Validation**: Quality checks and statistics generation ### Data Quality - ✅ Deduplicated using MD5 hash matching - ✅ Filtered entries < 2 characters - ✅ Removed rows with missing translations - ✅ Normalized whitespace - ✅ Validated Arabic and English text pairs ## Considerations for Using the Data ### Egyptian Arabic Characteristics Egyptian Arabic differs significantly from Modern Standard Arabic (MSA): - **Vocabulary**: Distinct colloquial words (e.g., إزيك vs. كيف حالك) - **Grammar**: Simplified structures (e.g., no case endings) - **Pronunciation**: Different phonetics (e.g., ج pronounced as "g") - **Script**: Informal spelling conventions in spoken contexts ### Recommended Training Approaches 1. **Fine-tune multilingual models** rather than training from scratch 2. **Combine with MSA data** for better Arabic understanding 3. **Use domain filtering** for specialized applications 4. **Consider episode context** for narrative tasks 5. **Balance domain distribution** if training general model ### Ethical Considerations - **Dialect Representation**: Egyptian Arabic is one of many Arabic dialects - **Cultural Context**: Translations maintain cultural nuances - **Source Attribution**: Data from TV series subtitles - **Privacy**: No personal information included ## License This dataset is released under the **CC BY 4.0 License**. ## Citation If you use this dataset in your research, please cite: ```bibtex @dataset{egyptian_dialogue_2026, title={Egyptian Arabic Dialogue Dataset}, author={fr3on}, year={2025}, publisher={Hugging Face}, url={https://huggingface.co/datasets/fr3on/egyptian-dialogue} } ``` ## Acknowledgments - Source: Egyptian TV series subtitles - Processing: Automatic domain detection and classification - Format: Parquet for efficaient loading and storage ## Version History - **v1.0.0** (2025-12-17): Initial release - 4,322 entries - 18 domain categories - Automatic domain detection - Parquet format --- **Keywords**: Egyptian Arabic, ar_EG, dialect, colloquial, translation, dialogue, domain classification, NLP, machine translation, Arabic dialects, conversational AI, parquet **Dataset Size**: 4,322 examples | **Format**: Parquet | **License**: CC BY 4.0language: - 阿拉伯语（ar） - 英语（en） license: CC BY 4.0 task_categories: - 机器翻译 - 文本生成 tags: - 埃及阿拉伯语 - 方言 - 口语体 - ar_EG - 机器翻译 - 对话 - 字幕 - 领域分类 pretty_name: 埃及阿拉伯语对话数据集 size_categories: - 1K<n<10K # 埃及阿拉伯语对话数据集 ## 数据集描述 本数据集包含**4322对埃及阿拉伯语-英语平行对话样本**，并附带自动领域分类标签。数据源自电视剧字幕，采用自然会话式埃及阿拉伯口语（العامية المصرية）。 ### 语言说明 - **源语言**：埃及阿拉伯语（ar_EG）——口语方言 - **目标语言**：英语（en） ## 数据集摘要 埃及阿拉伯语是使用最广泛的阿拉伯语方言之一，使用者超1亿。本数据集提供： - 自然会话式对话 - 口语表达与习语 - 用于专项训练的领域分类内容 - 用于叙事理解的剧集上下文 ## 数据集结构 ### 数据格式 每个样本包含： json { "id": "ep01_line0001", "arabic": "خلاويص؟", "english": "Ready or not?", "episode": 1, "dialect": "egyptian", "language": "ar", "language_variant": "ar_EG", "genre": "dialogue", "domain": "general" } ### 数据字段 | 字段名 | 类型 | 描述 | |-------|------|-------------| | `id` | 字符串 | 唯一标识符（格式：epXX_lineYYYY） | | `arabic` | 字符串 | 埃及阿拉伯语文本 | | `english` | 字符串 | 英语译文 | | `episode` | 整数 | 剧集编号（用于上下文关联） | | `dialect` | 字符串 | 方言标识（固定为"egyptian"） | | `language` | 字符串 | ISO语言代码（固定为"ar"） | | `language_variant` | 字符串 | 具体变体代码（固定为"ar_EG"） | | `genre` | 字符串 | 内容体裁（对话/旁白） | | `domain` | 字符串 | 自动检测的内容领域 | ## 数据集统计 ### 概览 - **总样本数**：4322 - **剧集数**：6 - **唯一领域数**：18 - **唯一体裁数**：2 - **埃及阿拉伯语文本平均长度**：25.9字符 - **英语译文平均长度**：35.0字符 ### 领域分布 | 领域 | 样本数 | 占比 | |--------|-------|------------| | 通用领域 | 2143 | 49.6% | | 科技 | 531 | 12.3% | | 家庭 | 368 | 8.5% | | 恐怖 | 281 | 6.5% | | 医疗 | 233 | 5.4% | | 爱情 | 136 | 3.1% | | 天气 | 115 | 2.7% | | 美食 | 104 | 2.4% | | 超自然 | 86 | 2.0% | | 社交 | 55 | 1.3% | ### 剧集分布 | 剧集 | 样本数 | |---------|---------| | 第1集 | 889 | | 第2集 | 782 | | 第3集 | 584 | | 第4集 | 907 | | 第5集 | 554 | | 第6集 | 606 | ### 体裁分布 - **对话**：4301（99.5%） - **旁白**：21（0.5%） ## 领域详解 本数据集采用基于关键词的检测方法实现**自动领域分类**，各领域说明如下： - **通用领域**：无特定主题的日常会话 - **家庭**：家庭关系、亲属、婚姻话题 - **恐怖**：惊悚主题、鬼魂、超自然恐惧 - **医疗**：医疗健康、医生、治疗相关 - **科技**：计算机、手机、互联网、应用程序 - **爱情**：恋爱、亲密关系、情感话题 - **超自然**：神秘、无法解释的现象 - **天气**：气候、气象、温度相关 - **美食**：烹饪、餐厅、餐食话题 - **社交**：朋友、聚会、社交生活 - **犯罪**：警察、调查、执法相关 - **教育**：学校、大学、学习相关 - **体育**：游戏、赛事、锦标赛 - **娱乐**：电影、剧集、影院相关 - **法律**：法律、法庭、法务相关 - **新闻**：新闻报道、媒体内容 - **商业**：公司、经济、贸易相关 - **政治**：政府、选举、政策相关 ## 应用场景 ### ✅ 推荐应用场景 - **埃及阿拉伯语翻译**：训练专门针对埃及方言的翻译模型 - **领域专属模型**：针对特定领域（医疗、法律等）训练模型 - **方言研究**：开展埃及阿拉伯语特征相关研究 - **会话式AI**：为埃及用户构建聊天机器人 - **语言建模**：针对埃及方言进行预训练或微调 - **多领域学习**：训练具备领域感知能力的模型 ### ⚠️ 局限性 - **领域范围**：仅覆盖娱乐/对话类内容 - **语体限制**：仅包含会话/非正式语言 - **样本规模**：共4322条样本，对于大规模预训练而言相对有限 - **方言变体**：埃及阿拉伯语存在未覆盖的区域次方言 - **上下文缺失**：单条对话样本可能缺乏完整叙事上下文 ## 数据集加载 ### 使用Hugging Face Datasets库 python from datasets import load_dataset # 加载数据集 dataset = load_dataset("fr3on/egyptian-dialogue") # 访问样本数据 print(dataset['train'][0]) # 按领域筛选 medical_data = dataset['train'].filter(lambda x: x['domain'] == 'medical') # 按剧集筛选 episode_1 = dataset['train'].filter(lambda x: x['episode'] == 1) ### 使用Pandas python import pandas as pd # 直接加载Parquet文件 df = pd.read_parquet("data/train-00000-of-00001.parquet") # 分析领域分布 print(df['domain'].value_counts()) # 筛选并导出 medical_df = df[df['domain'] == 'medical'] ## 训练示例 ### 翻译模型训练示例 python from datasets import load_dataset from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainer # 加载数据集 dataset = load_dataset("fr3on/egyptian-dialogue") # 加载阿拉伯语-英语翻译模型 model_name = "Helsinki-NLP/opus-mt-ar-en" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSeq2SeqLM.from_pretrained(model_name) # 预处理函数 def preprocess(examples): inputs = tokenizer(examples['arabic'], truncation=True, max_length=128) targets = tokenizer(examples['english'], truncation=True, max_length=128) inputs['labels'] = targets['input_ids'] return inputs tokenized = dataset.map(preprocess, batched=True) # 训练器配置 trainer = Seq2SeqTrainer( model=model, train_dataset=tokenized['train'], eval_dataset=tokenized['test'] ) trainer.train() ### 领域专属训练示例 python from datasets import load_dataset dataset = load_dataset("fr3on/egyptian-dialogue") # 按领域分别训练模型 for domain in ['medical', 'legal', 'technology']: domain_data = dataset['train'].filter(lambda x: x['domain'] == domain) # 训练领域专属模型 print(f"正在训练{domain}领域模型，共{len(domain_data)}条样本") ## 数据收集与处理 ### 数据来源 - **起源**：埃及电视剧字幕 - **语言**：专业字幕译文 - **语体**：自然会话式埃及阿拉伯语 ### 处理流程 1. **提取**：从Excel字幕文件中加载数据 2. **清洗**：移除空行、过短样本 3. **去重**：基于哈希的重复样本移除（共移除945条重复数据） 4. **领域检测**：基于关键词匹配实现自动分类 5. **体裁分类**：自动区分对话与旁白 6. **验证**：质量检查与统计信息生成 ### 数据质量 - ✅ 通过MD5哈希匹配实现去重 - ✅ 过滤掉长度小于2字符的样本 - ✅ 移除缺失译文的样本 - ✅ 标准化空白字符格式 - ✅ 验证阿拉伯语与英语文本配对的有效性 ## 数据使用注意事项 ### 埃及阿拉伯语特征 埃及阿拉伯语与现代标准阿拉伯语（MSA）存在显著差异： - **词汇**：独特的口语词汇（例如：إزيك 对应 现代标准阿拉伯语的 كيف حالك） - **语法**：简化的句法结构（例如：无格变化） - **发音**：不同的语音系统（例如：字母ج 发音为"g"） - **书写**：口语场景下的非正式拼写惯例 ### 推荐训练策略 1. **微调多语言模型**而非从零开始训练 2. **结合现代标准阿拉伯语数据**以提升整体阿拉伯语理解能力 3. **针对专项应用使用领域筛选** 4. **利用剧集上下文**开展叙事相关任务 5. **训练通用模型时平衡领域分布** ### 伦理考量 - **方言代表性**：埃及阿拉伯语仅是众多阿拉伯语方言之一 - **文化语境**：译文保留了文化细节 - **来源标注**：数据源自电视剧字幕 - **隐私保护**：未包含任何个人信息 ## 许可证 本数据集采用**CC BY 4.0许可证**发布。 ## 引用信息 如果您在研究中使用本数据集，请引用如下文献： bibtex @dataset{egyptian_dialogue_2026, title={Egyptian Arabic Dialogue Dataset}, author={fr3on}, year={2025}, publisher={Hugging Face}, url={https://huggingface.co/datasets/fr3on/egyptian-dialogue} } ## 致谢 - 数据来源：埃及电视剧字幕 - 处理流程：自动领域检测与分类 - 存储格式：Parquet，实现高效加载与存储 ## 版本历史 - **v1.0.0**（2025-12-17）：初始发布 - 4322条样本 - 18个领域分类 - 自动领域检测功能 - Parquet存储格式 --- **关键词**：埃及阿拉伯语、ar_EG、方言、口语体、机器翻译、对话、领域分类、自然语言处理（Natural Language Processing, NLP）、阿拉伯语方言、会话式AI、Parquet **数据集规模**：4322条样本 | **格式**：Parquet | **许可证**：CC BY 4.0