SharathReddy/Indian-Legal-SFT-Dataset

--- language: - en license: mit size_categories: - 10K<n<100K task_categories: - question-answering - text-generation tags: - legal - indian-law - instruction-tuning - vidhaan - constitution - justice configs: - config_name: default data_files: - split: train path: train.jsonl dataset_info: features: - name: instruction dtype: string description: "The legal question or task." - name: context dtype: string description: "The verbatim statutory text/section used for grounding." - name: response dtype: string description: "The professional legal answer with specific section citations." splits: - name: train num_examples: 20690 download_size: 8300000 dataset_size: 8300000 --- # Vidhaan: High-Density Indian Legal Instruction Dataset **Vidhaan** is a comprehensive, high-precision instruction-tuning dataset containing **20,690 QA pairs** derived from 113 Central Acts of India. It was built specifically to solve the "context-splitting" problem found in standard legal RAG datasets. ## 🛠 Dataset Structure & Format - **Primary File:** `vidhaan_training_v1.jsonl` - **Format:** JSON Lines (JSONL) - **Schema:** - `instruction`: (String) A precise legal query. - `context`: (String) The specific legal text or section header from the source file. - `response`: (String) A grounded answer starting with formal citations (e.g., "As per Section X..."). ## 📂 Source Composition The dataset spans 113 Markdown files organized into four primary domains: - **Constitution of India:** 1 Comprehensive File. - **Department of Justice:** 27 Files (e.g., Judges Inquiry Act, Family Courts Act). - **Department of Legal Affairs:** 11 Files (e.g., Advocates Act, Notaries Act). - **Legislative Department:** 74 Files (e.g., Indian Contract Act, Transfer of Property Act). ## 🧠 Methodology & Training Pipeline ### 1. High-Fidelity Conversion Original government PDFs were converted to Markdown using **Docling**. This ensured that critical structural elements like **State Amendment boxes** (e.g., Bihar/Assam specific changes) and **Footnotes** were captured as text rather than being discarded or mangled by standard OCR. ### 2. Semantic Logic Splitting To prevent a rule from being separated from its "Provided that" exception, we abandoned fixed-character chunking. We used **Regex-based Semantic Splitting** (`\n(?=\d+\.\s|##\s|CHAPTER\s)`) to ensure every training instance contains a complete, intact legal section. ### 3. Exhaustive QA Generation Using `gpt-4o-mini`, we performed **Exhaustive Content Mapping**. Instead of a fixed number of questions per chunk, the model was mandated to generate a pair for **every** distinct sub-section, definition, and procedural timeline found in the text. ## ✅ Quality Assurance & Validation - **Total Audited Pairs:** 20,690 - **Malformed/Skipped Lines:** 0 (Verified via post-processing audit). - **Section Coverage:** - **112/113 files:** Achieved **100% verified coverage** of all section headers. - **Code of Civil Procedure (CPC):** Achieved **98% coverage**. Note: The 2% "missing" were identified as false positives (years like 1870/1883 mentioned in text rather than missing section numbers). - **Citation Integrity:** 100% of responses contain verified statutory citations. ## 🚀 Use Cases - **Fine-tuning LLMs** for the Indian Judicial System. - **Evaluating Legal RAG** systems on statutory accuracy. - **Procedural Law Automation** (identifying limitation periods and appeal timelines). --- **Author:** Sharath Reddy **Project:** Vidhaan AI **Data Integrity:** Verified 100% string-type for all features.--- 语言： - 英语 许可证：MIT协议 规模类别： - 10000 < 样本数量 < 100000 任务类别： - 问答 - 文本生成 标签： - 法律 - 印度法律 - 指令微调（Instruction Tuning） - Vidhaan - 宪法 - 司法 配置： - 配置名称：默认配置 数据文件： - 拆分方式：训练集 文件路径：train.jsonl 数据集信息： 特征： - 名称：instruction 数据类型：字符串 描述：法律问题或任务 - 名称：context 数据类型：字符串 描述：用于锚定回答的逐字法定文本/条款 - 名称：response 数据类型：字符串 描述：带有具体条款引用的专业法律答复 拆分： - 拆分名称：训练集 样本数量：20690 下载大小：8300000 数据集大小：8300000 --- # Vidhaan：高密度印度法律指令数据集 **Vidhaan**是一款全面、高精度的指令微调（Instruction Tuning）数据集，包含源自印度113部中央法案的**20690个问答对**，其构建初衷是解决标准法律检索增强生成（Retrieval-Augmented Generation, RAG）数据集存在的“上下文割裂”问题。 ## 🛠 数据集结构与格式 - **主文件：** `vidhaan_training_v1.jsonl` - **格式：** JSON Lines（JSONL） - **数据结构：** - `instruction`：（字符串类型）精准的法律查询 - `context`：（字符串类型）来源文件中的特定法律文本或条款标题 - `response`：（字符串类型）以正式引用开头的锚定答复（例如“根据第X条……”） ## 📂 数据源构成 本数据集涵盖113个Markdown文件，划分为四大核心领域： - **《印度宪法》：** 1个完整文件 - **司法部：** 27个文件（例如《法官调查法》《家事法庭法》） - **法律事务部：** 11个文件（例如《律师法》《公证人法》） - **立法部：** 74个文件（例如《印度合同法》《财产转让法》） ## 🧠 方法论与训练管线 ### 1. 高保真转换 原始政府PDF文件通过**Docling**工具转换为Markdown格式，确保州修正案框（例如比哈尔邦/阿萨姆邦专属修改内容）和脚注等关键结构元素被完整保留为文本，而非被标准光学字符识别（Optical Character Recognition, OCR）丢弃或破坏。 ### 2. 语义逻辑拆分 为避免法律条文与其“但书”例外条款被拆分，我们放弃了固定字符分块策略，采用基于正则表达式的语义拆分方法（` (?=d+.s|##s|CHAPTERs)`），确保每个训练样本都包含完整、未割裂的法律条款。 ### 3. 全量问答生成 借助`gpt-4o-mini`，我们执行了**全量内容映射**。与为每个分块生成固定数量问答的方式不同，模型被要求为文本中每一个独立子条款、定义和程序时间线生成一组问答对。 ## ✅ 质量保证与验证 - **总审核问答对数量：** 20690 - **格式错误/跳过行数：** 0（经后处理审核确认） - **条款覆盖范围：** - 112/113个文件：实现所有条款标题的**100%验证覆盖** - 《民事诉讼法》（CPC）：实现**98%覆盖**。注：所谓“缺失”的2%实为误判，文本中提及的1870/1883等年份并非缺失的条款编号。 - **引用完整性：** 所有答复均包含经过验证的法定引用。 ## 🚀 应用场景 - 为印度司法系统**微调大语言模型（LLM）** - 针对法定准确性评估法律检索增强生成（RAG）系统 - 程序法自动化（识别时效期限与上诉时限） --- **作者：** Sharath Reddy **项目：** Vidhaan AI **数据完整性：** 所有特征均经验证为字符串类型。