feyninc/hojicha

--- language: - en license: cc-by-4.0 tags: - legal - contracts - chunking - rag - retrieval - nlp - cuad - mtcb pretty_name: Hojicha - Legal Contract Chunking Benchmark size_categories: - 1K<n<10K task_categories: - question-answering - text-retrieval dataset_info: - config_name: corpus features: - name: title dtype: string - name: text dtype: string splits: - name: train num_bytes: 8995933 num_examples: 194 download_size: 4035284 dataset_size: 8995933 - config_name: questions features: - name: question dtype: string - name: document_title dtype: string - name: chunk-must-contain dtype: string splits: - name: train num_bytes: 741445 num_examples: 1568 download_size: 305953 dataset_size: 741445 configs: - config_name: corpus data_files: - split: train path: corpus/train-* - config_name: questions data_files: - split: train path: questions/train-* --- # ⚖️ Hojicha - Legal Contract Chunking Benchmark **Hojicha** (HOldings JudIcial CHAllenges) is a benchmark dataset for evaluating text chunking algorithms on legal contracts. It is part of [MTCB (Make That Chunker Better)](https://github.com/chonkie-inc/mtcb). ## Dataset Description Hojicha tests how well chunking algorithms handle formal legal language, including: - **Nested clauses** and complex sentence structures - **Cross-references** between sections - **Legal terminology** and defined terms - **Structured contract sections** (recitals, definitions, covenants, etc.) ### Source Derived from [CUAD (Contract Understanding Atticus Dataset)](https://www.atticusprojectai.org/cuad), which contains commercial contracts annotated by legal experts. ### Statistics | Split | Count | |-------|-------| | Contracts | 479 | | Questions | 1,982 | | Question Types | 41 | ### Question Types The dataset covers 41 types of contract clauses, including: | Category | Examples | |----------|----------| | **Identification** | Document Name, Parties, Agreement Date, Effective Date | | **Term & Termination** | Expiration Date, Renewal Term, Termination for Convenience | | **Liability** | Cap on Liability, Uncapped Liability, Liquidated Damages | | **IP & Licensing** | License Grant, IP Ownership, Non-Transferable License | | **Restrictions** | Non-Compete, Exclusivity, Non-Solicitation, Anti-Assignment | | **Financial** | Revenue/Profit Sharing, Minimum Commitment, Price Restrictions | | **Other** | Governing Law, Insurance, Audit Rights, Change of Control | ## Usage ### With MTCB ```python from mtcb import HojichaEvaluator from chonkie import RecursiveChunker evaluator = HojichaEvaluator( chunker=RecursiveChunker(chunk_size=512), embedding_model="voyage-3-large", ) result = evaluator.evaluate(k=[1, 3, 5, 10]) print(result) ``` ### Direct Loading ```python from datasets import load_dataset # Load corpus (479 contracts) corpus = load_dataset("chonkie-ai/hojicha", "corpus", split="train") # Load questions (1,982 questions) questions = load_dataset("chonkie-ai/hojicha", "questions", split="train") ``` ## Data Format ### Corpus Each document contains: - `title`: Contract identifier (e.g., "COMPANY_DATE-EX-10-AGREEMENT TYPE") - `text`: Full contract text ### Questions Each question contains: - `question`: The question text (asking about a specific clause type) - `document_title`: Reference to the source contract - `chunk-must-contain`: The passage that must appear in retrieved chunks - `question_type`: Category of the clause (e.g., "Governing Law", "Cap On Liability") ## Evaluation Methodology For each question: 1. Chunk all contracts using the chunking algorithm 2. Embed all chunks and the question 3. Retrieve top-k chunks by similarity 4. Check if any retrieved chunk contains the `chunk-must-contain` passage 5. Calculate Recall@k and MRR@k ## License This dataset is released under CC-BY-4.0, following the original CUAD license. ## Citation ```bibtex @dataset{hojicha2025, title={Hojicha: Legal Contract Chunking Benchmark}, author={Chonkie Team}, year={2025}, publisher={Hugging Face}, url={https://huggingface.co/datasets/chonkie-ai/hojicha} } @inproceedings{cuad2021, title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review}, author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer}, booktitle={NeurIPS}, year={2021} } ``` ## Links - [MTCB GitHub](https://github.com/chonkie-inc/mtcb) - [Chonkie Chunking Library](https://github.com/chonkie-inc/chonkie) - [Original CUAD Dataset](https://www.atticusprojectai.org/cuad)--- 语言: - 英语 许可证: cc-by-4.0 标签: - 法律 - 合同 - 分块（Chunking） - 检索增强生成（Retrieval-Augmented Generation, RAG） - 检索 - 自然语言处理（Natural Language Processing, NLP） - CUAD - MTCB 展示名称: Hojicha——法律合同分块基准数据集 规模类别: - 1000 < 样本数 < 10000 任务类别: - 问答（Question-Answering） - 文本检索（Text-Retrieval） 数据集信息: - 配置名称: corpus 特征: - 名称: title 数据类型: string - 名称: text 数据类型: string 拆分: - 名称: train 字节数: 8995933 样本数: 194 下载大小: 4035284 数据集大小: 8995933 - 配置名称: questions 特征: - 名称: question 数据类型: string - 名称: document_title 数据类型: string - 名称: chunk-must-contain 数据类型: string 拆分: - 名称: train 字节数: 741445 样本数: 1568 下载大小: 305953 数据集大小: 741445 配置项: - 配置名称: corpus 数据文件: - 拆分: train 路径: corpus/train-* - 配置名称: questions 数据文件: - 拆分: train 路径: questions/train-* --- # ⚖️ Hojicha——法律合同分块基准数据集 **Hojicha**（全称HOldings JudIcial CHAllenges，即司法挑战持有基准）是用于评估法律合同文本分块（Chunking）算法的基准数据集，隶属于[MTCB（Make That Chunker Better，优化分块器基准）](https://github.com/chonkie-inc/mtcb)。 ## 数据集描述 Hojicha用于测试分块算法处理正式法律语言的能力，涵盖以下场景： - **嵌套条款（Nested Clauses）**与复杂句式结构 - **条款间交叉引用（Cross-References）** - **法律术语（Legal Terminology）与自定义术语** - **结构化合同章节（Structured Contract Sections）**（如引言、定义、契约条款等） ### 数据来源 本数据集衍生自[CUAD（合同理解阿提克斯数据集，Contract Understanding Atticus Dataset）](https://www.atticusprojectai.org/cuad)，该数据集包含由法律专家标注的商业合同。 ### 统计信息 | 拆分类型 | 数量 | |-------|-------| | 合同总数 | 479 | | 问题总数 | 1,982 | | 问题类型数 | 41 | ### 问题类型 本数据集覆盖41类合同条款，包括： | 类别 | 示例 | |----------|----------| | **身份识别类（Identification）** | 文档名称、合同主体、协议签署日期、生效日期 | | **条款与终止类（Term & Termination）** | 到期日期、续约期限、任意终止权 | | **责任类（Liability）** | 责任限额、无上限责任、违约金 | | **知识产权与许可类（IP & Licensing）** | 许可授予、知识产权归属、不可转让许可 | | **限制类（Restrictions）** | 竞业禁止、排他性条款、禁止挖角、禁止转让 | | **财务类（Financial）** | 收益/利润分配、最低承诺额、价格限制 | | **其他类（Other）** | 准据法、保险条款、审计权、控制权变更 | ## 使用方法 ### 结合MTCB使用 python from mtcb import HojichaEvaluator from chonkie import RecursiveChunker evaluator = HojichaEvaluator( chunker=RecursiveChunker(chunk_size=512), embedding_model="voyage-3-large", ) result = evaluator.evaluate(k=[1, 3, 5, 10]) print(result) ### 直接加载 python from datasets import load_dataset # 加载语料库（共479份合同） corpus = load_dataset("chonkie-ai/hojicha", "corpus", split="train") # 加载问题集（共1,982个问题） questions = load_dataset("chonkie-ai/hojicha", "questions", split="train") ## 数据格式 ### 语料库 每份文档包含： - `title`：合同标识符（例如："COMPANY_DATE-EX-10-AGREEMENT TYPE"） - `text`：完整合同文本 ### 问题集 每个问题包含： - `question`：问题文本（针对特定条款类型提问） - `document_title`：源合同的引用标识 - `chunk-must-contain`：检索分块时必须包含的段落 - `question_type`：条款类别（例如："Governing Law", "Cap On Liability"） ## 评估方法 针对每个问题，评估流程如下： 1. 使用分块算法对所有合同进行分块 2. 对所有分块与问题进行向量化编码 3. 基于相似度检索Top-k个分块 4. 检查任意检索得到的分块是否包含`chunk-must-contain`段落 5. 计算Recall@k（召回率@k）与MRR@k（平均倒数排名） ## 许可证 本数据集遵循CC-BY-4.0许可证发布，与原始CUAD数据集的许可证保持一致。 ## 引用格式 bibtex @dataset{hojicha2025, title={Hojicha: Legal Contract Chunking Benchmark}, author={Chonkie Team}, year={2025}, publisher={Hugging Face}, url={https://huggingface.co/datasets/chonkie-ai/hojicha} } @inproceedings{cuad2021, title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review}, author={Hendrycks, Dan and Burns, Collin and Chen, Anya and Ball, Spencer}, booktitle={NeurIPS}, year={2021} } ## 相关链接 - [MTCB GitHub仓库](https://github.com/chonkie-inc/mtcb) - [Chonkie分块库](https://github.com/chonkie-inc/chonkie) - [原始CUAD数据集](https://www.atticusprojectai.org/cuad)