Subh24ai/yojana-sahayak-instruct

--- language: - en - hi license: apache-2.0 task_categories: - text-generation - question-answering tags: - india - government-schemes - instruction-tuning - multilingual - hindi - yojana - welfare pretty_name: Yojana Sahayak Instruct size_categories: - 10K<n<100K --- # Yojana Sahayak Instruct Dataset ## Overview Instruction-tuning dataset for building an AI assistant that helps Indian citizens find and understand government welfare schemes in English and Hindi (Hinglish). Built as part of the **Yojana Sahayak** project — a multilingual voice assistant for Indian government schemes using fine-tuned LLMs + RAG + Whisper ASR. ## Dataset Stats | Split | Records | |-------|---------| | Train | ~31,965 | | Eval | ~7,992 | | **Total** | **~39,957** | | Language | Count | |----------|-------| | English | ~20,961 | | Hindi / Hinglish | ~18,996 | ## Data Source - Raw PDFs: [shrijayan/gov_myscheme](https://huggingface.co/datasets/shrijayan/gov_myscheme) (scraped from [myscheme.gov.in](https://www.myscheme.gov.in)) - 2,872 Indian government schemes across central and state governments ## Format Each record contains a `messages` list in chat format (system / user / assistant): ```json { "language": "hi", "scheme_name": "PM Kisan Samman Nidhi", "field": "eligibility", "messages": [ {"role": "system", "content": "You are Yojana Sahayak..."}, {"role": "user", "content": "PM Kisan ke liye kaun eligible hai?"}, {"role": "assistant", "content": "PM Kisan ke liye yeh log apply kar sakte hain: ..."} ] } ``` ## Fields Covered - `description` — what the scheme is - `eligibility` — who can apply - `benefits` — what you receive - `application_process` — how to apply - `multi_turn` — two-turn eligibility → benefits conversations ## Intended Use - Fine-tuning small LLMs (Qwen2.5-1.5B, Llama 3.2 1B) for scheme Q&A - Building RAG pipelines over scheme documents - Training multilingual chatbots for citizen services ## Usage ```python from datasets import load_dataset ds = load_dataset("Subh24ai/yojana-sahayak-instruct") print(ds["train"][0]) ``` ## License Apache 2.0. Original data sourced from Indian government's public MyScheme portal. ## Citation If you use this dataset, please cite the original MyScheme data source: > MyScheme, Government of India. https://www.myscheme.gov.in--- 语言： - 英语 - 印地语 许可证：Apache 2.0 任务类别： - 文本生成 - 问答 标签： - 印度 - 政府计划 - 指令微调 - 多语言 - 印地语 - 约杰纳（Yojana） - 福利 数据集名称：约杰纳助手指令数据集（Yojana Sahayak Instruct） 样本规模：10000 < 样本量 < 100000 --- # 约杰纳助手指令数据集（Yojana Sahayak Instruct） ## 概述 本数据集为指令微调数据集，用于构建可帮助印度民众以英语、印地语（混合印地语英语，即Hinglish）查找并理解政府福利计划的AI智能体。 本数据集是**约杰纳助手（Yojana Sahayak）**项目的组成部分，该项目旨在打造一款基于微调大语言模型（LLM）、检索增强生成（RAG）以及Whisper自动语音识别（ASR）的印度政府计划多语言语音助手。 ## 数据集统计 | 拆分 | 样本量 | |-------|---------| | 训练集 | ~31965 | | 评估集 | ~7992 | | **总计** | **~39957** | | 语言 | 样本量 | |----------|-------| | 英语 | ~20961 | | 印地语/混合印地语英语 | ~18996 | ## 数据来源 - 原始PDF数据：[shrijayan/gov_myscheme](https://huggingface.co/datasets/shrijayan/gov_myscheme)（从[myscheme.gov.in](https://www.myscheme.gov.in)爬取） - 涵盖中央与地方政府推出的共计2872项印度政府福利计划 ## 数据格式 每条数据均包含聊天格式的`messages`列表，包含系统（system）、用户（user）与助手（assistant）三类角色： json { "language": "hi", "scheme_name": "PM Kisan Samman Nidhi", "field": "eligibility", "messages": [ {"role": "system", "content": "You are Yojana Sahayak..."}, {"role": "user", "content": "PM Kisan ke liye kaun eligible hai?"}, {"role": "assistant", "content": "PM Kisan ke liye yeh log apply kar sakte hain: ..."} ] } ## 覆盖字段 - `description`：计划详情介绍 - `eligibility`：申请资格要求 - `benefits`：可享受的福利内容 - `application_process`：申请流程 - `multi_turn`：两轮对话格式的资格要求→福利内容问答场景 ## 预期用途 - 针对政府计划问答场景，微调小型大语言模型（如Qwen2.5-1.5B、Llama 3.2 1B） - 基于政府计划文档构建检索增强生成（RAG）流程 - 训练面向民众服务的多语言聊天机器人 ## 使用方法 python from datasets import load_dataset ds = load_dataset("Subh24ai/yojana-sahayak-instruct") print(ds["train"][0]) ## 许可证 Apache 2.0。原始数据源自印度政府官方MyScheme门户。 ## 引用说明 若使用本数据集，请引用原始MyScheme数据源： > MyScheme, 印度政府. https://www.myscheme.gov.in