alikhan126/loato-bench-artifacts

--- license: mit task_categories: - text-classification language: - en tags: - prompt-injection - security - embeddings - loato - capstone size_categories: - 10K<n<100K --- # LOATO-Bench Artifacts Pre-computed embeddings, experiment results, and dataset files for the **LOATO-Bench** project — studying cross-attack generalization of embedding-based prompt injection classifiers. **GitHub repo**: [alikhan126/loato-bench](https://github.com/alikhan126/loato-bench) ## What's in this repo | Path | Size | Description | |------|------|-------------| | `embeddings/minilm/` | 107 MB | all-MiniLM-L6-v2 (384d) embeddings for 68,845 samples | | `embeddings/bge_large/` | 283 MB | BGE-large-en-v1.5 (1024d) embeddings | | `embeddings/instructor/` | 283 MB | Instructor-large (1024d) embeddings | | `embeddings/openai_small/` | 424 MB | text-embedding-3-small (1536d) embeddings | | `embeddings/e5_mistral/` | 1.1 GB | E5-Mistral-7B GGUF Q4 (4096d) embeddings | | `results/experiments/` | 256 KB | 30 experiment result JSONs (5 models × 3 classifiers × 2 protocols) | | `data/processed/labeled_v1.parquet` | 11 MB | Final labeled dataset (68,845 samples: 40,017 benign / 28,828 injection) | | `data/processed/unified_dataset.parquet` | 11 MB | Pre-labeling harmonized dataset | | `data/splits/` | 6 MB | Train/test split indices for all 4 evaluation protocols | ## Why this exists The embedding step takes **10+ hours** to run from scratch (E5-Mistral alone takes ~8 hours on an M3 Pro). By hosting pre-computed artifacts here, anyone can reproduce the full experiment pipeline in minutes instead of hours. ## How to use ### Option 1: Download script (recommended) From the [loato-bench](https://github.com/alikhan126/loato-bench) repo: ```bash # Set your HF token in .env echo "HF_TOKEN=your_token_here" >> .env # Download everything (~2.2 GB) uv run python scripts/download_artifacts.py # Or download selectively uv run python scripts/download_artifacts.py --only embeddings uv run python scripts/download_artifacts.py --only results uv run python scripts/download_artifacts.py --only data ``` ### Option 2: Python API ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="alikhan126/loato-bench-artifacts", repo_type="dataset", local_dir="./artifacts", token="your_token_here", # or set HF_TOKEN env var ) ``` ### Option 3: Git clone ```bash git lfs install git clone https://huggingface.co/datasets/alikhan126/loato-bench-artifacts ``` ## Embedding format Each embedding is stored as a compressed NumPy file (`.npz`): ```python import numpy as np data = np.load("embeddings/minilm/embeddings.npz") embeddings = data["embeddings"] # shape: (68845, dim) sample_ids = data["sample_ids"] # shape: (68845,) ``` The `meta.json` sidecar contains model name, dimensions, sample count, and a text hash for cache validation. ## Experiment results format Each JSON file contains per-fold metrics (F1, accuracy, AUC-ROC, precision, recall) for a specific embedding × classifier × experiment combination: ``` results/experiments/{experiment}_{embedding}_{classifier}.json ``` Example: `loato_e5_mistral_mlp.json` = E5-Mistral embeddings + MLP classifier under LOATO protocol. ## Dataset 68,845 samples from 9 public sources: - **Injection** (28,828): Open-Prompt-Injection, HackAPrompt, PINT/Gandalf, Deepset - **Benign** (40,017): Dolly 15K, Alpaca (cleaned), OASST1, WildChat (nontoxic) Labeled with a 3-tier taxonomy (source maps → regex → GPT-4o-mini) into 7 attack categories. ## License MIT — Academic use (Pace University MS Data Science Capstone).--- 许可证：MIT 任务类别： - 文本分类 语言： - 英语 标签： - 提示注入（prompt-injection） - 安全 - 嵌入（embeddings） - loato - 顶石项目（capstone） 规模类别： - 10K<n<100K --- # LOATO-Bench 工件 本仓库包含**LOATO-Bench**项目的预计算嵌入、实验结果与数据集文件——该项目旨在研究基于嵌入的提示注入分类器的跨攻击泛化能力。 **GitHub 仓库**：[alikhan126/loato-bench](https://github.com/alikhan126/loato-bench) ## 本仓库内容 | 路径 | 大小 | 描述 | |------|------|-------------| | `embeddings/minilm/` | 107 MB | 针对68,845个样本的all-MiniLM-L6-v2（384维）嵌入 | | `embeddings/bge_large/` | 283 MB | BGE-large-en-v1.5（1024维）嵌入 | | `embeddings/instructor/` | 283 MB | Instructor-large（1024维）嵌入 | | `embeddings/openai_small/` | 424 MB | text-embedding-3-small（1536维）嵌入 | | `embeddings/e5_mistral/` | 1.1 GB | E5-Mistral-7B GGUF Q4（4096维）嵌入 | | `results/experiments/` | 256 KB | 30份实验结果JSON文件（5种模型 × 3种分类器 × 2种协议） | | `data/processed/labeled_v1.parquet` | 11 MB | 最终标注数据集（68,845个样本：40,017个良性样本 / 28,828个注入样本） | | `data/processed/unified_dataset.parquet` | 11 MB | 预标注统一化数据集 | | `data/splits/` | 6 MB | 针对全部4种评估协议的训练/测试集划分索引 | ## 本仓库的意义 嵌入计算步骤从零开始运行需耗时10小时以上（仅E5-Mistral在M3 Pro设备上就需约8小时）。通过在此托管预计算的工件，任何人都可将完整实验流程的复现时间从数小时缩短至数分钟。 ## 使用方式 ### 选项1：下载脚本（推荐） 从[loato-bench](https://github.com/alikhan126/loato-bench)仓库执行： bash # 在.env文件中设置你的Hugging Face令牌 echo "HF_TOKEN=your_token_here" >> .env # 下载全部内容（约2.2 GB） uv run python scripts/download_artifacts.py # 或选择性下载 uv run python scripts/download_artifacts.py --only embeddings uv run python scripts/download_artifacts.py --only results uv run python scripts/download_artifacts.py --only data ### 选项2：Python API python from huggingface_hub import snapshot_download snapshot_download( repo_id="alikhan126/loato-bench-artifacts", repo_type="dataset", local_dir="./artifacts", token="your_token_here", # 或设置HF_TOKEN环境变量 ) ### 选项3：Git克隆 bash git lfs install git clone https://huggingface.co/datasets/alikhan126/loato-bench-artifacts ## 嵌入格式 每个嵌入文件以压缩NumPy文件（`.npz`）格式存储： python import numpy as np data = np.load("embeddings/minilm/embeddings.npz") embeddings = data["embeddings"] # 形状：(68845, dim) sample_ids = data["sample_ids"] # 形状：(68845,) 附带的`meta.json`元数据文件包含模型名称、维度、样本数量以及用于缓存验证的文本哈希值。 ## 实验结果格式 每个JSON文件包含针对特定嵌入×分类器×实验组合的每折指标（F1值、准确率、AUC-ROC、精确率、召回率）： results/experiments/{experiment}_{embedding}_{classifier}.json 示例：`loato_e5_mistral_mlp.json` = 采用E5-Mistral嵌入 + MLP分类器，基于LOATO协议的实验结果。 ## 数据集说明 本数据集包含来自9个公开来源的68,845个样本： - **注入样本（28,828个）**：Open-Prompt-Injection、HackAPrompt、PINT/Gandalf、Deepset - **良性样本（40,017个）**：Dolly 15K、清理版Alpaca、OASST1、非毒性WildChat 数据集采用三级分类体系（来源映射 → 正则表达式 → GPT-4o-mini）标注为7种攻击类别。 ## 许可证 MIT许可证——学术用途（佩斯大学数据科学硕士顶石课程项目）。