mjbommar/opengloss-v1.3-definitions

--- license: cc-by-4.0 task_categories: - text-generation - question-answering - text-classification - feature-extraction language: - en tags: - dictionary - lexicon - wordnet - semantic-network - knowledge-graph - encyclopedic - etymology - synthetic - education size_categories: - 100K<n<1M --- # OpenGloss Dictionary v1.3 (Definition-Level) ## Dataset Summary **OpenGloss** is a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. This dataset provides the **definitions-level view** where each record represents one sense definition. ### Key Statistics - **565,604 sense definitions** - **8,479,875 semantic edges** (synonyms, antonyms, hypernyms, hyponyms, collocations, inflections) - **205,983 entries** with encyclopedic content (100.0% coverage) - **194,420 entries** with etymology (94.4% coverage) - **149,734 entries** with Wikipedia frequency data (72.7% coverage) - **100% reading level coverage** (K through PhD scale) - **100% domain tag coverage** (10+ subject domain categories) - **Average 2.75 senses per lexeme** - **Average 41.2 edges per lexeme** ### What's New in v1.3? Compared to OpenGloss v1.2: 1. **Expanded lexicon coverage**: more lexeme records and more definition-level records in the base dictionary exports 2. **Hard negative pairs dataset**: a new calibration-oriented dataset for embedding training and score separation 3. **Larger companion datasets**: expanded query examples, contrastive examples, and encyclopedia variants in the release family 4. **Gap-driven coverage expansion**: broader geography, history, civics, and related weak-domain support carried into the release 5. **Unified release family**: dictionary, definitions, query, contrastive, encyclopedia, and hard-negative datasets aligned under one version ### POS Distribution | Part of Speech | Count | |----------------|-------| | noun | 165,258 | | adjective | 65,477 | | verb | 39,532 | | adverb | 7,121 | | determiner | 1,511 | | preposition | 1,237 | | interjection | 974 | | pronoun | 397 | | conjunction | 251 | | particle | 19 | | proper noun | 13 | | numeral | 5 | | proper_noun | 4 | | prefix | 2 | | suffix | 1 | | abbreviation | 1 | | adjetivo | 1 | | sustantivo | 1 | ### Edge Type Distribution | Relationship Type | Count | |-------------------|-------| | synonym | 1,651,142 | | collocation | 1,453,581 | | hyponym | 1,317,685 | | hypernym | 1,109,943 | | antonym | 1,036,163 | | etymology_parent | 882,355 | | inflection | 378,445 | | derivation_noun | 279,502 | | derivation_adjective | 163,037 | | derivation_verb | 119,944 | | derivation_adverb | 68,106 | | cognate | 19,972 | ## Loading the Dataset ```python from datasets import load_dataset # Load the full dataset dataset = load_dataset("mjbommar/opengloss-v1.3-definitions") # Access records for record in dataset["train"]: print(f"Word: {record['word']}") print(f"Senses: {record['total_senses']}") print(f"Edges: {record['total_edges']}\n") ``` ## Core Fields & Usage Examples ### Wikipedia Frequency Data Filter by word importance using frequency data: ```python # Get high-frequency words (top 10,000) common_words = dataset["train"].filter( lambda x: x["wiki_frequency_rank"] is not None and x["wiki_frequency_rank"] <= 10000 ) # Sort by frequency sorted_by_freq = dataset["train"].sort("wiki_frequency", reverse=True) ``` ### Reading Levels Filter vocabulary by grade level for educational applications: ```python # Elementary (K-5) elementary = dataset["train"].filter(lambda x: x["reading_level"] in ["K", "1", "2", "3", "4", "5"]) # Middle school (6-8) middle_school = dataset["train"].filter(lambda x: x["reading_level"] in ["6", "7", "8"]) # High school (9-12) high_school = dataset["train"].filter(lambda x: x["reading_level"] in ["9", "10", "11", "12"]) # Advanced (BS/PhD) advanced = dataset["train"].filter(lambda x: x["reading_level"] in ["BS", "PhD"]) ``` ### Domain Tags Filter by subject area for content-specific applications: ```python # Science vocabulary science_words = dataset["train"].filter( lambda x: any("science" in tag or "life-sciences" in tag for tag in x.get("tags", [])) ) # Technology vocabulary tech_words = dataset["train"].filter( lambda x: any("technology" in tag for tag in x.get("tags", [])) ) # Social studies social_studies = dataset["train"].filter( lambda x: any(tag.startswith("domain:history") or tag.startswith("domain:society") for tag in x.get("tags", [])) ) ``` ### Etymology Segments Access structured etymology with language trail: ```python # Words with detailed etymology words_with_etymology = dataset["train"].filter(lambda x: len(x.get("etymology_segments", [])) > 0) # Find words from specific language origins latin_origin = dataset["train"].filter( lambda x: any(seg.get("language", "").lower() == "latin" for seg in x.get("etymology_segments", [])) ) ``` ## Citation If you use OpenGloss in your research, please cite: ```bibtex @misc{bommarito2025opengloss, title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-v1.3-definitions}, note={Dataset available under CC-BY 4.0} } ``` ## License This dataset is released under **Creative Commons Attribution 4.0 International (CC-BY 4.0)**. ## Version History - **v1.3** (2026-04): ~206K entries with gap-fill expansion, regenerated lexical explanations with relation data, full companion dataset coverage for top 50K entries, multiple encyclopedia variants - **v1.2** (2026-04): Expanded release with larger companion training datasets and hard-negative calibration pairs - **v1.1** (2025-11): Release with structured morphology, etymology segments, and frequency data - **v1.0** (2025-01): Initial release ## Acknowledgments This dataset was generated using: - [pydantic-ai](https://github.com/pydantic/pydantic-ai) for structured LLM generation - OpenAI GPT models for content generation - Anthropic Claude for quality assurance --- *Generated from the OpenGloss v1.3 dataset.*--- license: cc-by-4.0 任务类别： - 文本生成 - 问答 - 文本分类 - 特征提取 语言： - 英语 标签： - 词典 - 词表（Lexicon） - 词网（WordNet） - 语义网络（Semantic Network） - 知识图谱（Knowledge Graph） - 百科全书式 - 词源学（Etymology） - 合成式 - 教育 数据规模： - 10万 < 样本数 < 100万 --- # OpenGloss 词典 v1.3（定义层级） ## 数据集概述 **OpenGloss** 是一款面向英语的合成式百科词典与语义知识图谱（Knowledge Graph），将词典编纂释义、百科背景信息、词源历史与语义关系整合为统一资源。 本数据集提供**定义层级视图**，每条记录对应一个义项释义。 ## 核心统计数据 - **565,604 个义项释义** - **8,479,875 条语义边（同义词、反义词、上位词（Hypernym）、下位词（Hyponym）、搭配、屈折形式（Inflection）） - **205,983 个词条带有百科内容（覆盖度100.0%） - **194,420 个词条带有词源信息（覆盖度94.4%） - **149,734 个词条带有维基百科频率数据（覆盖度72.7%） - **100% 阅读水平覆盖（从幼儿园至博士层级） - **100% 领域标签覆盖（包含10+个主题领域分类） - **每个词形平均拥有2.75个义项** - **每个词形平均拥有41.2条语义边** ## v1.3版本更新内容 相较于OpenGloss v1.2版本： 1. **扩展词表覆盖：基础词典导出文件中包含更多词形记录与更多定义层级记录 2. **难负样本数据集：面向嵌入训练与分数分离的新型校准导向数据集 3. **更大规模配套数据集：本次发布套件中包含扩展的查询示例、对比示例与百科变体数据集 4. **缺口驱动式覆盖扩展：将更广泛的地理、历史、公民与相关弱领域支持纳入本次发布 5. **统一发布套件：词典、定义、查询、对比、百科与难负样本数据集均统一至同一版本下 ## 词性分布 | 词性 | 数量 | |------|------| | 名词 | 165,258 | | 形容词 | 65,477 | | 动词 | 39,532 | | 副词 | 7,121 | | 限定词 | 1,511 | | 介词 | 1,237 | | 感叹词 | 974 | | 代词 | 397 | | 连词 | 251 | | 小品词 | 19 | | 专有名词 | 13 | | 数词 | 5 | | 专有名词（proper_noun） | 4 | | 前缀 | 2 | | 后缀 | 1 | | 缩写 | 1 | | 形容词（adjetivo） | 1 | | 名词（sustantivo） | 1 | ## 关系类型分布 | 关系类型 | 数量 | |----------|------| | 同义词 | 1,651,142 | | 搭配 | 1,453,581 | | 下位词 | 1,317,685 | | 上位词 | 1,109,943 | | 反义词 | 1,036,163 | | 词源父项 | 882,355 | | 屈折形式 | 378,445 | | 名词派生词 | 279,502 | | 形容词派生词 | 163,037 | | 动词派生词 | 119,944 | | 副词派生词 | 68,106 | | 同源词 | 19,972 | ## 数据集加载 python from datasets import load_dataset # 加载完整数据集 dataset = load_dataset("mjbommar/opengloss-v1.3-definitions") # 访问记录 for record in dataset["train"]: print(f"词汇：{record["word"]}") print(f"义项总数：{record["total_senses"]}") print(f"语义边总数：{record["total_edges"]} ") ## 核心字段与使用示例 ### 维基百科频率数据 使用频率数据筛选词汇重要性： python # 获取高频词汇（前10000名 common_words = dataset["train"].filter( lambda x: x["wiki_frequency_rank"] is not None and x["wiki_frequency_rank"] <= 10000 ) # 按频率排序 sorted_by_freq = dataset["train"].sort("wiki_frequency", reverse=True) ### 阅读水平分级 针对教育应用场景按年级筛选词汇： python # 小学（K-5年级） elementary = dataset["train"].filter(lambda x: x["reading_level"] in ["K", "1", "2", "3", "4", "5"]) # 中学（6-8年级） middle_school = dataset["train"].filter(lambda x: x["reading_level"] in ["6", "7", "8"]) # 高中（9-12年级） high_school = dataset["train"].filter(lambda x: x["reading_level"] in ["9", "10", "11", "12"]) # 进阶（本科/博士） advanced = dataset["train"].filter(lambda x: x["reading_level"] in ["BS", "PhD"]) ### 领域标签筛选 针对特定主题应用按领域标签筛选： python # 科学词汇 science_words = dataset["train"].filter( lambda x: any("science" in tag or "life-sciences" in tag for tag in x.get("tags", [])) ) # 技术词汇 tech_words = dataset["train"].filter( lambda x: any("technology" in tag for tag in x.get("tags", [])) ) # 社会研究 social_studies = dataset["train"].filter( lambda x: any(tag.startswith("domain:history") or tag.startswith("domain:society") for tag in x.get("tags", [])) ) ### 词源片段 通过语言溯源获取结构化词源信息： python # 获取带有详细词源的词汇 words_with_etymology = dataset["train"].filter(lambda x: len(x.get("etymology_segments", [])) > 0) # 查找特定语源的词汇 latin_origin = dataset["train"].filter( lambda x: any(seg.get("language", "").lower() == "latin" for seg in x.get("etymology_segments", [])) ) ## 引用信息 如果您在研究中使用OpenGloss，请引用以下文献： bibtex @misc{bommarito2025opengloss, title={OpenGloss: 一款合成式百科词典与语义知识图谱}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-v1.3-definitions}, note={数据集采用CC-BY 4.0许可协议} } ## 许可协议 本数据集采用**知识共享署名4.0国际许可协议（CC-BY 4.0）**。 ## 版本历史 - **v1.3**（2026-04）：约20.6万个词条，通过缺口填充扩展，重新生成带关系数据的词法解释，覆盖前5万个词条的完整配套数据集，包含多个百科变体 - **v1.2**（2026-04）：扩展发布，包含更大规模的配套训练数据集与难负样本校准对 - **v1.1**（2025-11：发布包含结构化词法、词源片段与频率数据 - **v1.0**（2025-01：初始发布 ## 致谢 本数据集通过以下工具生成： - [pydantic-ai](https://github.com/pydantic/pydantic-ai) 用于结构化大语言模型（Large Language Model/LLM）生成 - OpenAI GPT模型用于内容生成 - Anthropic Claude用于质量保证 *本内容源自OpenGloss v1.3数据集。