nassimjp/bactrian_ps

--- dataset_info: features: - name: instruction dtype: string - name: input dtype: string - name: id dtype: string - name: output dtype: string splits: - name: train num_bytes: 72372209 num_examples: 67017 download_size: 36656858 dataset_size: 72372209 configs: - config_name: default data_files: - split: train path: data/train-* --- # Dataset Card for Bactrian-X-Pashto This is a specialized, Pashto-only subset of the **Bactrian-X** dataset, optimized into a native Parquet format. ## 1. Why this version was created The original **MBZUAI/Bactrian-X** dataset relies on a `Bactrian-X.py` loading script. Due to recent security updates on the Hugging Face Hub, **dataset scripts are no longer supported or executed by default**, often resulting in the following error during training: > `RuntimeError: Dataset scripts are no longer supported, but found Bactrian-X.py` To ensure uninterrupted research and compatibility with modern libraries like `unsloth`, `trl`, and `peft`, this version was created by extracting the Pashto data and converting it into a **script-free Apache Parquet format**. This allows researchers to load the data directly without security flags or custom trust overrides. ## 2. Dataset Description - **Language:** Pashto (ps_AF) - **Format:** Parquet (High-performance, no loading script required) - **Optimization:** Designed for SFT/QLoRA training on high-end hardware (e.g., RTX 4070 Ti Super / 3090). ### Summary A collection of **67,017** instruction-response pairs obtained by translating English Alpaca and Dolly instructions into Pashto via Google Translate, with responses generated by `gpt-3.5-turbo`. ## 3. Dataset Structure The data follows the standard instruction-tuning schema: - **`instruction`**: The task (e.g., "توضیح ورکړئ..."). - **`input`**: Contextual data (if applicable). - **`output`**: The target response. - **`id`**: Original reference ID. ## 4. Technical Details - **Source:** Extracted from the `ps` split of MBZUAI/Bactrian-X. - **Conversion:** Converted to Parquet to solve the `.py` script execution block. - **Compatibility:** Fully compatible with `load_dataset("nassimjp/bactrian_ps")` without requiring `trust_remote_code=True`. ## 5. Additional Information ### Licensing Available under **Creative Commons NonCommercial (CC BY-NC 4.0)**. ### Citation ```bibtex @misc{li2023bactrianx, title={Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation}, author={Haonan Li and Fajri Koto and Minghao Wu and Alham Fikri Aji and Timothy Baldwin}, year={2023}, eprint={2305.15011}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` ### Curators - **Original:** Haonan Li and Fajri Koto. - **Parquet Migration:** nassimjp --- ## 🚀 How to Load Since this version is stored in the native Hugging Face Parquet format, you can load it in one line without worrying about security flags or external `.py` scripts: ```python from datasets import load_dataset # Load the Pashto subset directly dataset = load_dataset("nassimjp/bactrian_ps", split="train") # Example of accessing the first row print(dataset[0]) ``` ### Advantages of this version: 1. **No `trust_remote_code=True` required:** Bypasses the `RuntimeError: Dataset scripts are no longer supported` error. 2. **Performance:** Parquet files are significantly faster to stream and load compared to the original JSON-plus-script method. 3. **Clean Schema:** Includes `instruction`, `input`, and `output` columns ready for immediate use in SFT trainers. ------ dataset_info: 数据集信息： 特征： - 字段名：instruction，数据类型：字符串（string） - 字段名：input，数据类型：字符串（string） - 字段名：id，数据类型：字符串（string） - 字段名：output，数据类型：字符串（string） 划分集： - 划分名称：train，字节数：72372209，样本数：67017 下载大小：36656858 数据集总大小：72372209 配置项： - 配置名称：default，数据文件： - 划分：train，路径：data/train-* --- # Bactrian-X-普什图语 数据集卡片 本数据集为**Bactrian-X**数据集的专用普什图语单语子集，已优化为原生Parquet格式（Apache Parquet）。 ## 1. 本版本的创建缘由 原版**MBZUAI/Bactrian-X**数据集依赖`Bactrian-X.py`加载脚本。鉴于Hugging Face Hub（Hugging Face 枢纽平台）近期的安全更新，**数据集脚本默认不再受支持或执行**，这常会在训练阶段触发如下错误： > `RuntimeError: Dataset scripts are no longer supported, but found Bactrian-X.py` 为保障研究过程不受干扰，并兼容`unsloth`、`trl`、`peft`等现代机器学习库，本版本通过提取普什图语数据并转换为**无脚本Apache Parquet格式（Apache Parquet）**生成。研究人员可直接加载该数据集，无需设置安全标记或自定义信任覆盖项。 ## 2. 数据集说明 - **语言**：普什图语（ps_AF） - **格式**：Parquet格式（Apache Parquet，高性能，无需加载脚本） - **优化方向**：针对高端硬件（如RTX 4070 Ti Super/3090）的监督微调（SFT, Supervised Fine-Tuning）/量化低秩适配器微调（QLoRA, Quantized Low-Rank Adaptation）训练设计。 ### 数据集概览 本数据集包含**67017条**指令-回复对：通过谷歌翻译将英文Alpaca数据集（Alpaca）与Dolly数据集（Dolly）的指令转换为普什图语，回复由`gpt-3.5-turbo`生成。 ## 3. 数据集结构 本数据集遵循标准的指令微调范式： - **`instruction`**：任务描述（例如："توضیح ورکړئ..."，即普什图语的“请解释...”）。 - **`input`**：上下文数据（如适用）。 - **`output`**：目标回复。 - **`id`**：原始引用ID。 ## 4. 技术细节 - **数据来源**：从MBZUAI/Bactrian-X的`ps`划分集中提取。 - **格式转换**：转换为Parquet格式以解决`.py`脚本执行限制问题。 - **兼容性**：可直接通过`load_dataset("nassimjp/bactrian_ps")`加载，无需设置`trust_remote_code=True`。 ## 5. 附加信息 ### 授权协议 本数据集采用**知识共享非商业授权（CC BY-NC 4.0）**。 ### 引用 bibtex @misc{li2023bactrianx, title={Bactrian-X : A Multilingual Replicable Instruction-Following Model with Low-Rank Adaptation}, author={Haonan Li and Fajri Koto and Minghao Wu and Alham Fikri Aji and Timothy Baldwin}, year={2023}, eprint={2305.15011}, archivePrefix={arXiv}, primaryClass={cs.CL} } ### 数据集维护者 - **原始版本**：Haonan Li与Fajri Koto。 - **Parquet格式迁移**：nassimjp --- ## 🚀 加载方式 由于本数据集采用Hugging Face原生Parquet格式存储，您可通过一行代码直接加载，无需担心安全标记或外部`.py`脚本问题： python from datasets import load_dataset # 直接加载普什图语子集 dataset = load_dataset("nassimjp/bactrian_ps", split="train") # 访问第一条数据的示例 print(dataset[0]) ### 本版本的优势： 1. **无需设置`trust_remote_code=True`**：可规避`RuntimeError: Dataset scripts are no longer supported`错误。 2. **性能优异**：相较于原版的JSON加脚本加载方式，Parquet格式文件的流式读取与加载速度显著提升。 3. **范式规范**：包含`instruction`、`input`与`output`字段，可直接用于监督微调训练器。 ---