mjbommar/opengloss-v1.2-definitions

--- license: cc-by-4.0 task_categories: - text-generation - question-answering - text-classification - feature-extraction language: - en tags: - dictionary - lexicon - wordnet - semantic-network - knowledge-graph - encyclopedic - etymology - synthetic - education size_categories: - 100K<n<1M --- # OpenGloss Dictionary v1.2 (Definition-Level) ## Dataset Summary **OpenGloss** is a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. This dataset provides the **definitions-level view** where each record represents one sense definition. ### Key Statistics - **517,892 sense definitions** - **7,798,653 semantic edges** (synonyms, antonyms, hypernyms, hyponyms, collocations, inflections) - **162,314 entries** with encyclopedic content (100.0% coverage) - **150,746 entries** with etymology (92.9% coverage) - **149,734 entries** with Wikipedia frequency data (92.2% coverage) - **100% reading level coverage** (K through PhD scale) - **100% domain tag coverage** (10+ subject domain categories) - **Average 3.19 senses per lexeme** - **Average 48.0 edges per lexeme** ### What's New in v1.2? Compared to OpenGloss v1.1: 1. **Expanded lexicon coverage**: more lexeme records and more definition-level records in the base dictionary exports 2. **Hard negative pairs dataset**: a new calibration-oriented dataset for embedding training and score separation 3. **Larger companion datasets**: expanded query examples, contrastive examples, and encyclopedia variants in the release family 4. **Gap-driven coverage expansion**: broader geography, history, civics, and related weak-domain support carried into the release 5. **Unified release family**: dictionary, definitions, query, contrastive, encyclopedia, and hard-negative datasets aligned under one version ### POS Distribution | Part of Speech | Count | |----------------|-------| | noun | 134,241 | | adjective | 55,955 | | verb | 36,423 | | adverb | 5,583 | | determiner | 1,510 | | preposition | 1,234 | | interjection | 941 | | pronoun | 395 | | conjunction | 249 | | particle | 18 | | proper noun | 12 | | numeral | 5 | | proper_noun | 4 | | prefix | 2 | | suffix | 1 | | adjetivo | 1 | | sustantivo | 1 | | abbreviation | 1 | ### Edge Type Distribution | Relationship Type | Count | |-------------------|-------| | synonym | 1,512,031 | | hyponym | 1,285,609 | | collocation | 1,273,658 | | hypernym | 1,018,466 | | antonym | 1,007,062 | | etymology_parent | 697,921 | | inflection | 353,345 | | derivation_noun | 279,502 | | derivation_adjective | 163,037 | | derivation_verb | 119,944 | | derivation_adverb | 68,106 | | cognate | 19,972 | ## Loading the Dataset ```python from datasets import load_dataset # Load the full dataset dataset = load_dataset("mjbommar/opengloss-v1.2-definitions") # Access records for record in dataset["train"]: print(f"Word: {record['word']}") print(f"Senses: {record['total_senses']}") print(f"Edges: {record['total_edges']}\n") ``` ## Core Fields & Usage Examples ### Wikipedia Frequency Data Filter by word importance using frequency data: ```python # Get high-frequency words (top 10,000) common_words = dataset["train"].filter( lambda x: x["wiki_frequency_rank"] is not None and x["wiki_frequency_rank"] <= 10000 ) # Sort by frequency sorted_by_freq = dataset["train"].sort("wiki_frequency", reverse=True) ``` ### Reading Levels Filter vocabulary by grade level for educational applications: ```python # Elementary (K-5) elementary = dataset["train"].filter(lambda x: x["reading_level"] in ["K", "1", "2", "3", "4", "5"]) # Middle school (6-8) middle_school = dataset["train"].filter(lambda x: x["reading_level"] in ["6", "7", "8"]) # High school (9-12) high_school = dataset["train"].filter(lambda x: x["reading_level"] in ["9", "10", "11", "12"]) # Advanced (BS/PhD) advanced = dataset["train"].filter(lambda x: x["reading_level"] in ["BS", "PhD"]) ``` ### Domain Tags Filter by subject area for content-specific applications: ```python # Science vocabulary science_words = dataset["train"].filter( lambda x: any("science" in tag or "life-sciences" in tag for tag in x.get("tags", [])) ) # Technology vocabulary tech_words = dataset["train"].filter( lambda x: any("technology" in tag for tag in x.get("tags", [])) ) # Social studies social_studies = dataset["train"].filter( lambda x: any(tag.startswith("domain:history") or tag.startswith("domain:society") for tag in x.get("tags", [])) ) ``` ### Etymology Segments Access structured etymology with language trail: ```python # Words with detailed etymology words_with_etymology = dataset["train"].filter(lambda x: len(x.get("etymology_segments", [])) > 0) # Find words from specific language origins latin_origin = dataset["train"].filter( lambda x: any(seg.get("language", "").lower() == "latin" for seg in x.get("etymology_segments", [])) ) ``` ## Citation If you use OpenGloss in your research, please cite: ```bibtex @misc{bommarito2025opengloss, title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-v1.2-definitions}, note={Dataset available under CC-BY 4.0} } ``` ## License This dataset is released under **Creative Commons Attribution 4.0 International (CC-BY 4.0)**. ## Version History - **v1.2** (2026-04): Expanded release with larger companion training datasets and hard-negative calibration pairs - **v1.1** (2025-11): Release with structured morphology, etymology segments, and frequency data - **v1.0** (2025-01): Initial release ## Acknowledgments This dataset was generated using: - [pydantic-ai](https://github.com/pydantic/pydantic-ai) for structured LLM generation - OpenAI GPT models for content generation - Anthropic Claude for quality assurance --- *Generated from the OpenGloss v1.2 dataset.*许可证：CC-BY-4.0 任务类别： - 文本生成 - 问答 - 文本分类 - 特征抽取 语言： - 英语 标签： - 词典 - 词表 - 词网（wordnet） - 语义网络（semantic-network） - 知识图谱（knowledge-graph） - 百科类 - 词源学 - 合成式 - 教育 样本规模分类： - 10万<n<100万 # OpenGloss词典v1.2（定义层级版） ## 数据集概述 **OpenGloss（OpenGloss）**是一款面向英语的合成式百科词典与语义知识图谱，将词典释义、百科背景信息、词源历史及语义关系整合为统一资源。 本数据集提供**定义层级视图**，每条记录对应一个义项释义。 ### 关键统计指标 - **517,892 个义项释义** - **7,798,653 条语义边**（涵盖同义词、反义词、上位词、下位词、搭配、屈折变化） - **162,314 条带百科内容的条目**（覆盖率100.0%） - **150,746 条带词源信息的条目**（覆盖率92.9%） - **149,734 条带维基百科频率数据的条目**（覆盖率92.2%） - **100% 阅读水平覆盖**（覆盖从K到PhD的分级体系） - **100% 领域标签覆盖**（涵盖10余个主题领域类别） - **每个词位平均3.19个义项** - **每个词位平均48.0条语义边** ### v1.2版本更新说明 相较于OpenGloss v1.1，本次更新包含以下内容： 1. **扩展词表覆盖范围**：基础词典导出文件中新增了更多词位记录与义项层级记录 2. **难负样本对数据集**：面向嵌入训练与分数分离任务的全新校准数据集 3. **更大规模配套数据集**：发布套件中新增了扩展查询示例、对比示例与百科变体数据集 4. **基于缺口的覆盖扩展**：进一步覆盖了地理、历史、公民学及相关弱领域内容 5. **统一化发布套件**：将词典、定义、查询、对比、百科及难负样本数据集整合为统一版本体系 ### 词性分布 | 词性 | 数量 | |------|------| | 名词（noun） | 134,241 | | 形容词（adjective） | 55,955 | | 动词（verb） | 36,423 | | 副词（adverb） | 5,583 | | 限定词（determiner） | 1,510 | | 介词（preposition） | 1,234 | | 感叹词（interjection） | 941 | | 代词（pronoun） | 395 | | 连词（conjunction） | 249 | | 小品词（particle） | 18 | | 专有名词（proper noun） | 12 | | 数词（numeral） | 5 | | 专有名词（proper_noun） | 4 | | 前缀（prefix） | 2 | | 后缀（suffix） | 1 | | adjetivo（西班牙语：形容词） | 1 | | sustantivo（西班牙语：名词） | 1 | | 缩写词（abbreviation） | 1 | ### 语义边类型分布 | 关系类型 | 数量 | |----------|------| | 同义词（synonym） | 1,512,031 | | 下位词（hyponym） | 1,285,609 | | 搭配（collocation） | 1,273,658 | | 上位词（hypernym） | 1,018,466 | | 反义词（antonym） | 1,007,062 | | 词源父项（etymology_parent） | 697,921 | | 屈折变化（inflection） | 353,345 | | 名词派生（derivation_noun） | 279,502 | | 形容词派生（derivation_adjective） | 163,037 | | 动词派生（derivation_verb） | 119,944 | | 副词派生（derivation_adverb） | 68,106 | | 同源词（cognate） | 19,972 | ## 数据集加载 python from datasets import load_dataset # 加载完整数据集 dataset = load_dataset("mjbommar/opengloss-v1.2-definitions") # 访问数据集记录 for record in dataset["train"]: print(f"单词：{record['word']}") print(f"义项总数：{record['total_senses']}") print(f"语义边总数：{record['total_edges']} ") ## 核心字段与使用示例 ### 维基百科频率数据 通过频率数据筛选词汇重要性： python # 获取高频词汇（前10000个） common_words = dataset["train"].filter( lambda x: x["wiki_frequency_rank"] is not None and x["wiki_frequency_rank"] <= 10000 ) # 按频率降序排序 sorted_by_freq = dataset["train"].sort("wiki_frequency", reverse=True) ### 阅读水平分级 针对教育应用按年级筛选词汇： python # 小学阶段（K-5年级） elementary = dataset["train"].filter(lambda x: x["reading_level"] in ["K", "1", "2", "3", "4", "5"]) # 初中阶段（6-8年级） middle_school = dataset["train"].filter(lambda x: x["reading_level"] in ["6", "7", "8"]) # 高中阶段（9-12年级） high_school = dataset["train"].filter(lambda x: x["reading_level"] in ["9", "10", "11", "12"]) # 高等教育阶段（学士/博士） advanced = dataset["train"].filter(lambda x: x["reading_level"] in ["BS", "PhD"]) ### 领域标签 按主题领域筛选特定应用场景的词汇： python # 科学领域词汇 science_words = dataset["train"].filter( lambda x: any("science" in tag or "life-sciences" in tag for tag in x.get("tags", [])) ) # 技术领域词汇 tech_words = dataset["train"].filter( lambda x: any("technology" in tag for tag in x.get("tags", [])) ) # 社会研究领域 social_studies = dataset["train"].filter( lambda x: any(tag.startswith("domain:history") or tag.startswith("domain:society") for tag in x.get("tags", [])) ) ### 词源片段 访问带语源追踪的结构化词源信息： python # 带有详细词源信息的词汇 words_with_etymology = dataset["train"].filter(lambda x: len(x.get("etymology_segments", [])) > 0) # 查找拉丁语源的词汇 latin_origin = dataset["train"].filter( lambda x: any(seg.get("language", "").lower() == "latin" for seg in x.get("etymology_segments", [])) ) ## 引用 如果您在研究中使用OpenGloss，请引用以下文献： bibtex @misc{bommarito2025opengloss, title={OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph}, author={Bommarito, Michael J., II}, year={2025}, url={https://huggingface.co/datasets/mjbommar/opengloss-v1.2-definitions}, note={Dataset available under CC-BY 4.0} } ## 许可证 本数据集采用**知识共享署名4.0国际许可协议（CC-BY-4.0）**发布。 ## 版本历史 - **v1.2（2026-04）**：扩展版发布，包含更大规模的配套训练数据集与难负样本校准对 - **v1.1（2025-11）**：发布了带结构化词形变化、词源片段与频率数据的版本 - **v1.0（2025-01）**：初始发布版本 ## 致谢 本数据集通过以下工具生成： - [pydantic-ai](https://github.com/pydantic/pydantic-ai)：用于结构化大语言模型（LLM）生成 - OpenAI GPT系列模型：用于内容生成 - Anthropic Claude：用于质量校验 --- *本内容基于OpenGloss v1.2数据集生成。*