mjbommar/opengloss-dictionary

--- license: cc-by-4.0 task_categories: - text-generation - question-answering - text-classification - feature-extraction language: - en tags: - dictionary - lexicon - wordnet - semantic-network - knowledge-graph - encyclopedic - etymology - synthetic - education size_categories: - 100K<n<1M --- # OpenGloss Dictionary (Word-Level) ## Dataset Summary **OpenGloss** is a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. This dataset provides the **words-level view** where each record represents one lexeme (word or multi-word expression). ### Key Statistics - **150,101 lexemes** across 150,101 English lexemes - **9.1 million semantic edges** (synonyms, antonyms, hypernyms, hyponyms, collocations, inflections) - **1 million usage examples** demonstrating words in context - **3 million collocations** showing common word combinations - **60 million words of encyclopedic content** (200-400 words per entry, 99.7% coverage) - **Etymology trails** for 97.5% of entries documenting historical development - **Average 3.58 senses per lexeme**, balancing granularity with usability ### What Makes OpenGloss Unique? Unlike traditional computational lexicons: 1. **Integrated Content**: Each entry combines definitions, examples, semantic relationships, morphology, collocations, encyclopedic context, and etymology 2. **Pedagogical Focus**: Designed for K-12 education and vocabulary learning with age-appropriate content 3. **Rich Connectivity**: Near-universal semantic relationship coverage (99.7% of senses have synonyms, hypernyms, and examples) 4. **Multi-word Expressions**: 37.3% of lexemes are multi-word phrases reflecting natural language usage 5. **Synthetic Generation**: Created via multi-agent LLM pipeline with schema validation in <1 week for <$1,000 ## Dataset Structure ### Data Format This dataset is provided as JSONL (JSON Lines), with each line containing one complete record. ### Word-Level Schema Each record represents a complete lexeme entry with all its senses. **Core Fields:** - `id`: Unique identifier for the lexeme - `word`: The lexeme string (e.g., "algorithm", "a bit") - `text`: Full markdown rendering of the entire entry (optional field) **Lexical Information:** - `parts_of_speech`: List of POS tags (e.g., ["noun", "verb"]) - `total_senses`: Total number of sense definitions - `sense_count_by_pos`: Breakdown of senses per POS - `senses`: Array of all sense definitions with their semantic relationships **Semantic Relationships:** - `all_synonyms`: Flattened list of all synonyms across senses - `all_antonyms`: Flattened list of all antonyms - `all_hypernyms`: Broader concepts (ordered specific → general) - `all_hyponyms`: Narrower concepts - `all_collocations`: Common multi-word expressions - `edges`: Complete graph edges (9.1M total across dataset) **Morphology:** - `all_inflections`: Inflected forms (plurals, tenses, etc.) - `all_derivations`: Morphologically derived forms **Enrichment:** - `has_etymology`: Boolean flag - `etymology_summary`: Historical development narrative - `etymology_cognates`: Cross-linguistic cognates - `has_encyclopedia`: Boolean flag - `encyclopedia_entry`: 200-400 word contextual explanation **Metadata:** - `is_stopword`: Boolean classification - `stopword_reason`: Explanation if stopword - `processed_at`: ISO timestamp ### Example Record ```json { "id": "algorithm", "word": "algorithm", "text": "# algorithm\n\n## Part of Speech: noun\n\n### Sense 1: ...", "parts_of_speech": ["noun"], "total_senses": 2, "senses": [ { "part_of_speech": "noun", "sense_index": 0, "definition": "A finite, stepwise procedure for solving a problem or completing a computation.", "synonyms": ["procedure", "process", "method", "routine"], "antonyms": [], "hypernyms": ["procedure", "technique", "system"], "hyponyms": ["sorting algorithm", "search algorithm"], "examples": [ "The student traced each algorithm step to verify the answer.", "We compared an arithmetic algorithm with a geometric approach." ] } ], "all_synonyms": ["procedure", "process", "method", "routine"], "all_hypernyms": ["procedure", "technique", "system"], "all_collocations": ["algorithm design", "sorting algorithm"], "all_inflections": ["algorithms"], "all_derivations": ["algorithmic", "algorithmically"], "has_etymology": true, "etymology_summary": "From Arabic 'al-Khwarizmi', named after 9th-century Persian mathematician...", "has_encyclopedia": true, "encyclopedia_entry": "An algorithm is a systematic procedure that defines a finite sequence of well-defined operations...", "is_stopword": false, "edges": [...], "total_edges": 47 } ``` ### Use Cases This **word-level dataset** is ideal for: - **Dictionary/Thesaurus Applications**: Complete lexeme lookup - **Vocabulary Learning**: Integrated definitions, examples, and encyclopedic context - **Semantic Search**: Rich metadata for filtering and ranking - **Knowledge Graph Construction**: Pre-structured semantic relationships - **Reading Comprehension Tools**: Contextual explanations for learners - **NLP Feature Extraction**: Multi-dimensional lexical features ## Dataset Creation ### Generation Methodology OpenGloss was created using a **multi-agent procedural generation pipeline** with: 1. **Lexeme Selection**: 150,101 lexemes from American English word lists + educational vocabulary expansion 2. **Sense Generation**: Two-agent architecture (overview + POS details) producing schema-validated definitions 3. **Graph Construction**: Deterministic edge extraction creating 9.1M semantic relationships 4. **Enrichment**: Etymology and encyclopedia agents adding contextual content All outputs use Pydantic V2 schema validation ensuring structural consistency. ### Models and Infrastructure - **Generation**: OpenAI GPT-5-nano via pydantic-ai - **Quality Assurance**: Claude Sonnet 4.5 - **Cost**: <$1,000 total API spend - **Time**: <96 hours wall-clock time - **Validation**: 100% edge target validity, automatic retry on malformed outputs ### Quality Characteristics **Strengths:** - Comprehensive coverage (99.7% encyclopedia, 97.5% etymology) - Consistent schema and formatting - Rich semantic connectivity (avg 17 edges per sense) - Integrated multi-dimensional content - Rapid iteration capability **Limitations:** - **Synthetic generation**: Reflects LLM training data patterns and biases - **Not expert-validated**: Unlike manually curated resources - **Potential inaccuracies**: Especially in technical domains and etymology - **Contemporary bias**: May lack historical usage nuances - **Schema constraints**: Fixed relationship types may miss subtle semantic distinctions ### Appropriate Use Cases ✅ **Recommended for:** - Educational technology and vocabulary learning - Rapid prototyping of lexical applications - Semantic feature extraction for NLP - Benchmark dataset for definition generation - Resource augmentation (combining with other datasets) - Research on synthetic knowledge resources ⚠️ **Use with caution for:** - Authoritative reference (verify critical information) - Fine-grained semantic analysis requiring expert validation - Historical linguistics research (etymology is plausible but not scholarly) - Domain-specific terminology (may lack precision) ## Comparison with Other Resources | Resource | Senses | Lexemes | Multi-word | Encyclopedic | Etymology | Cost | Update Cycle | |----------|--------|---------|------------|--------------|-----------|------|--------------| | **OpenGloss** | **537K** | **150K** | **37.3%** | **99.7%** | **97.5%** | **<$1K** | **<1 week** | | WordNet 3.1 | 117K | 155K | ~30% | ✗ | ✗ | Manual | Years | | Open English WordNet | 120K | 147K | ~30% | ✗ | ✗ | Manual | Ongoing | | BabelNet | 23M | 23M | Yes | Partial | ✗ | Integration | Ongoing | | ConceptNet | ~1.5M | ~800K | Yes | ✗ | ✗ | Crowdsourced | Ongoing | OpenGloss provides **4.6× more sense definitions** than WordNet while adding encyclopedic and etymological content absent from computational lexicons. **Overlap Analysis:** - OpenGloss ∩ WordNet: 38% vocabulary overlap - Each contributes distinct lexicographic priorities - OpenGloss emphasizes pedagogical vocabulary and multi-word expressions - Complementary rather than redundant coverage ## Loading the Dataset ```python from datasets import load_dataset # Load the full dataset dataset = load_dataset("mjbommar/opengloss-dictionary") # Access records for record in dataset["train"]: print(f"Word: {record['word']}") print(f"Definition: {record['definition'] if 'definition' in record else record['senses'][0]['definition']}") print(f"Edges: {record.get('total_edges', len(record.get('edges', [])))}\n") ``` ### Filtering Examples ```python # Filter by part of speech nouns = dataset["train"].filter(lambda x: parts_of_speech: "noun") # Find highly polysemous words polysemous = dataset["train"].filter( lambda x: x.get("total_senses", x.get("total_senses_for_word", 0)) >= 5 ) # Get entries with encyclopedic content with_encyclopedia = dataset["train"].filter(lambda x: x["has_encyclopedia"]) ``` ## Citation If you use OpenGloss in your research, please cite: ```bibtex @misc{bommarito2025opengloss, title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-dictionary}, note={Dataset available under CC-BY 4.0} } ``` ## License This dataset is released under **Creative Commons Attribution 4.0 International (CC-BY 4.0)**. You are free to: - **Share**: Copy and redistribute the material - **Adapt**: Remix, transform, and build upon the material Under the following terms: - **Attribution**: You must give appropriate credit and indicate if changes were made ## Additional Resources - 📄 **Paper**: Full methodology and analysis (available on arXiv) - 💾 **Alternative View**: [Definition-level dataset](https://huggingface.co/datasets/mjbommar/opengloss-dictionary-definitions) - 🔗 **Source Code**: [Generation pipeline](https://github.com/mjbommar/opengloss) (if applicable) - 📊 **Statistics**: See paper Section 4 for detailed dataset statistics ## Version History - **v1.0** (2025-01): Initial release - 150,101 lexemes, 536,829 senses - 9.1M semantic edges - 99.7% encyclopedic coverage, 97.5% etymology coverage ## Acknowledgments This dataset was generated using: - [pydantic-ai](https://github.com/pydantic/pydantic-ai) for structured LLM generation - OpenAI GPT-5-nano for content generation - Anthropic Claude Sonnet 4.5 for quality assurance Portions of this work were prepared with assistance from large language models. The author is solely responsible for all content, including any errors or omissions. ## Contact For questions, issues, or feedback: - **Email**: [e-mail] - **Dataset Issues**: Use the Hugging Face dataset discussion board --- *Generated from the OpenGloss v1.0 dataset. Last updated: 2025-01*许可证：CC-BY-4.0 任务类别： - 文本生成 - 问答 - 文本分类 - 特征抽取 语言： - 英语 标签： - 词典 - 词库 - WordNet - 语义网络 - 知识图谱 - 百科类 - 词源学 - 合成式 - 教育 规模分类： - 10万<n<100万 # OpenGloss 词典（词级版本） ## 数据集摘要 **OpenGloss** 是一款合成式英语百科词典与语义知识图谱（semantic knowledge graph），将词典释义、百科语境、词源历史与语义关系整合为统一的资源。本数据集提供**词级视角**，每条记录对应一个词位（lexeme），即单词或多词表达式。 ### 核心统计数据 - **150,101个词位**，覆盖150,101个英语词位 - **910万条语义边**，涵盖同义词、反义词、上位词、下位词、搭配与屈折形式 - **100万条语境用例**，展示单词的实际使用场景 - **300万条搭配**，呈现常见的单词组合 - **6000万词的百科内容**，单条条目长度为200-400词，覆盖率达99.7% - **97.5%的条目包含词源追踪**，记录词汇的历史发展脉络 - **每个词位平均含3.58个义项**，兼顾粒度与易用性 ### OpenGloss的独特优势 相较于传统计算词典： 1. **内容整合性**：每条条目同时包含释义、用例、语义关系、词形变化、搭配、百科语境与词源信息 2. **教学导向性**：专为K12教育与词汇学习设计，内容适配适龄学习者 3. **丰富的语义连接**：实现近乎全覆盖的语义关系（99.7%的义项包含同义词、上位词与用例） 4. **支持多词表达式**：37.3%的词位为多词短语，贴合自然语言的实际使用习惯 5. **合成式生成**：通过多智能体大语言模型（Large Language Model, LLM）流水线生成，附带Schema验证，总耗时不足1周，总成本低于1000美元 ## 数据集结构 ### 数据格式 本数据集以JSONL（JSON Lines）格式提供，每行对应一条完整的记录。 ### 词级Schema 每条记录代表一个完整的词位条目及其所有义项。 **核心字段**： - `id`：词位的唯一标识符 - `word`：词位字符串（例如："algorithm"，"a bit"） - `text`：完整的Markdown渲染条目（可选字段） **词汇信息**： - `parts_of_speech`：词性标签列表（例如：["noun", "verb"]） - `total_senses`：义项总数 - `sense_count_by_pos`：按词性划分的义项分布 - `senses`：包含所有义项定义及其语义关系的数组 **语义关系**： - `all_synonyms`：所有义项同义词的扁平化列表 - `all_antonyms`：所有义项反义词的扁平化列表 - `all_hypernyms`：上位概念（按从具体到通用的顺序排列） - `all_hyponyms`：下位概念 - `all_collocations`：常见多词表达式 - `edges`：完整的图边（数据集总共有910万条） **词形变化**： - `all_inflections`：屈折形式（复数、时态等） - `all_derivations`：词形派生形式 **内容丰富项**： - `has_etymology`：布尔标记，指示是否包含词源信息 - `etymology_summary`：词源发展历史叙述 - `etymology_cognates`：跨语言同源词 - `has_encyclopedia`：布尔标记，指示是否包含百科内容 - `encyclopedia_entry`：200-400词的语境化解释 **元数据**： - `is_stopword`：布尔分类标记，指示是否为停用词 - `stopword_reason`：若为停用词的原因说明 - `processed_at`：ISO格式时间戳 ### 示例记录 json { "id": "algorithm", "word": "algorithm", "text": "# algorithm ## Part of Speech: noun ### Sense 1: ...", "parts_of_speech": ["noun"], "total_senses": 2, "senses": [ { "part_of_speech": "noun", "sense_index": 0, "definition": "A finite, stepwise procedure for solving a problem or completing a computation.", "synonyms": ["procedure", "process", "method", "routine"], "antonyms": [], "hypernyms": ["procedure", "technique", "system"], "hyponyms": ["sorting algorithm", "search algorithm"], "examples": [ "The student traced each algorithm step to verify the answer.", "We compared an arithmetic algorithm with a geometric approach." ] } ], "all_synonyms": ["procedure", "process", "method", "routine"], "all_hypernyms": ["procedure", "technique", "system"], "all_collocations": ["algorithm design", "sorting algorithm"], "all_inflections": ["algorithms"], "all_derivations": ["algorithmic", "algorithmically"], "has_etymology": true, "etymology_summary": "From Arabic 'al-Khwarizmi', named after 9th-century Persian mathematician...", "has_encyclopedia": true, "encyclopedia_entry": "An algorithm is a systematic procedure that defines a finite sequence of well-defined operations...", "is_stopword": false, "edges": [...], "total_edges": 47 } ### 应用场景 这款**词级数据集**非常适合： - **词典/词库应用**：完整词位查询 - **词汇学习**：整合的释义、用例与百科语境 - **语义搜索**：用于过滤与排序的丰富元数据 - **知识图谱构建**：预结构化的语义关系 - **阅读理解工具**：为学习者提供语境化解释 - **自然语言处理特征抽取**：多维度的词汇特征 ## 数据集创建 ### 生成方法 OpenGloss采用**多智能体流程生成流水线**创建，具体步骤如下： 1. **词位筛选**：从美式英语词表与教育词汇扩展集中选取150,101个词位 2. **义项生成**：采用双智能体架构（概述+词性细节）生成符合Schema验证的义项定义 3. **图谱构建**：通过确定性边提取生成910万条语义关系 4. **内容丰富**：由词源与百科智能体补充语境内容 所有输出均使用Pydantic V2 Schema验证，确保结构一致性。 ### 模型与基础设施 - **内容生成**：基于pydantic-ai的OpenAI GPT-5-nano - **质量保证**：Anthropic Claude Sonnet 4.5 - **成本**：总API开销低于1000美元 - **耗时**：实际运行时间不足96小时 - **验证**：100%的边目标有效性，对格式错误的输出自动重试 ### 质量特性 **优势**： - 全面的覆盖率（99.7%百科内容，97.5%词源信息） - 统一的Schema与格式 - 丰富的语义连接（每个义项平均17条边） - 整合的多维度内容 - 快速迭代能力 **局限性**： - **合成式生成**：反映大语言模型训练数据的模式与偏差 - **未经专家验证**：不同于经人工甄选审核的资源 - **潜在 inaccuracies**：尤其是在技术领域与词源信息中可能存在不准确之处 - **当代偏差**：可能缺乏历史使用细节 - **Schema约束**：固定的关系类型可能遗漏细微的语义区别 ### 适用场景 ✅ **推荐使用场景**： - 教育技术与词汇学习 - 词汇应用的快速原型开发 - 自然语言处理的语义特征抽取 - 释义生成任务的基准数据集 - 资源增强（与其他数据集结合使用） - 合成式知识资源相关研究 ⚠️ **谨慎使用场景**： - 权威参考资料（需验证关键信息） - 需要专家验证的细粒度语义分析 - 历史语言学研究（词源信息虽合理但非学术级） - 特定领域术语（可能缺乏精确性） ## 与其他资源的对比 | 资源 | 义项数 | 词位数 | 多词表达式占比 | 百科内容 | 词源信息 | 成本 | 更新周期 | |----------|--------|---------|------------|--------------|-----------|------|--------------| | **OpenGloss** | **53.7万** | **15万** | **37.3%** | **99.7%** | **97.5%** | **<$1000** | **<1周** | | WordNet 3.1 | 11.7万 | 15.5万 | ~30% | ✗ | ✗ | 人工编纂 | 数年 | | Open English WordNet | 12万 | 14.7万 | ~30% | ✗ | ✗ | 人工编纂 | 持续更新 | | BabelNet | 2300万 | 2300万 | 是 | 部分 | ✗ | 整合式 | 持续更新 | | ConceptNet | ~150万 | ~80万 | 是 | ✗ | ✗ | 众包 | 持续更新 | OpenGloss的义项数量是WordNet的4.6倍，同时补充了传统计算词典所缺失的百科与词源内容。 **重叠分析**： - OpenGloss与WordNet的词汇重叠率为38% - 二者各有侧重的词典编纂优先级 - OpenGloss强调教学词汇与多词表达式 - 二者为互补而非重复覆盖 ## 数据集加载 python from datasets import load_dataset # 加载完整数据集 dataset = load_dataset("mjbommar/opengloss-dictionary") # 访问记录 for record in dataset["train"]: print(f"单词: {record['word']}") print(f"释义: {'definition' in record and record['definition'] or record['senses'][0]['definition']}") print(f"边数: {record.get('total_edges', len(record.get('edges', [])))} ") ### 筛选示例 python # 按词性筛选 nouns = dataset["train"].filter(lambda x: "noun" in x["parts_of_speech"]) # 查找多义项单词（义项数≥5） polysemous = dataset["train"].filter( lambda x: x.get("total_senses", x.get("total_senses_for_word", 0)) >=5 ) # 获取包含百科内容的条目 with_encyclopedia = dataset["train"].filter(lambda x: x["has_encyclopedia"]) ## 引用 如果您在研究中使用OpenGloss，请引用如下文献： bibtex @misc{bommarito2025opengloss, title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-dictionary}, note={数据集采用CC-BY 4.0协议发布} } ## 许可证 本数据集采用**知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International, CC-BY 4.0）**发布。 您可以自由： - **共享**：复制并分发本材料 - **改编**：修改、转换并基于本材料进行创作 需遵守以下条款： - **署名**：您必须给予适当的署名，并说明是否对材料进行了修改 ## 额外资源 - 📄 **学术论文**：完整的方法学与分析（可在arXiv获取） - 💾 **替代版本**：[义级数据集](https://huggingface.co/datasets/mjbommar/opengloss-dictionary-definitions) - 🔗 **源代码**：[生成流水线](https://github.com/mjbommar/opengloss)（如有） - 📊 **统计数据**：详见论文第4节的详细数据集统计 ## 版本历史 - **v1.0**（2025-01）：初始发布 - 150,101个词位，536,829个义项 - 910万条语义边 - 99.7%的百科内容覆盖率，97.5%的词源信息覆盖率 ## 致谢 本数据集的生成使用了： - [pydantic-ai](https://github.com/pydantic/pydantic-ai) 用于结构化大语言模型生成 - OpenAI GPT-5-nano 用于内容生成 - Anthropic Claude Sonnet 4.5 用于质量保证 本工作部分借助大型语言模型完成。作者对所有内容（包括任何错误或疏漏）承担全部责任。 ## 联系方式 如有疑问、问题或反馈，请联系： - **邮箱**：[e-mail] - **数据集问题**：使用Hugging Face数据集讨论板块 --- *本内容基于OpenGloss v1.0数据集生成。最后更新：2025-01*