TELOS XSTest Over-Refusal Calibration Dataset

Validation of TELOS governance framework against XSTest (NAACL 2024), measuring over-refusal rates with domain-specific Primacy Attractor calibration. Key Results: - Generic Safety PA: 24.80% over-refusal rate (62/250 safe prompts incorrectly flagged) - Healthcare HIPAA PA: 8.00% over-refusal rate (20/250 safe prompts incorrectly flagged) - Calibration Improvement: 16.80 percentage points reduction This demonstrates the core TELOS value proposition: domain-specific PA calibration reduces false positives while maintaining safety guarantees. Benchmark: XSTest (NAACL 2024) Source: https://github.com/paul-rottger/exaggerated-safety Paper: "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models" Dataset Contents: - xstest_validation_results.json: Generic Safety PA validation (250 safe prompts, 200 unsafe control) - xstest_healthcare_validation_results.json: Healthcare HIPAA PA validation (250 safe prompts) - general_safety_pa_config.json: Generic PA configuration used - healthcare_hipaa_pa_config.json: Healthcare PA configuration use Embedding Model: Sentence-transformers/all-MiniLM-L6-v2 (384-dim). Lightweight model demonstrates framework efficacy; results expected to improve with higher-capacity embeddings. License: Apache 2.0