howellx/diegetic-enterprise-training-data

---
license: mit
task_categories:
- text-generation
- question-answering
language:
- en
tags:
- diegetic
- epistemic-ai
- rag
- compliance
- grounding
- hallucination-prevention
- enterprise
size_categories:
- 100K<n<1M
---

# DIEGETIC Enterprise Training Data

Training dataset for building **epistemically-constrained AI systems** — models that only claim what they can justify from evidence, cite sources, and refuse when uncertain.

## Dataset Overview

| File | Examples | Description |
|------|----------|-------------|
| `enterprise_v1_sft.jsonl` | 113,934 | Supervised fine-tuning examples |
| `enterprise_v1_dpo.jsonl` | 113,934 | DPO preference pairs (grounded vs. hallucinated) |
| `enterprise_v1_microtasks.jsonl` | 34,431 | Belief update microtasks |

## Generation Method

Generated from 10,000 synthetic trajectories across 5 epistemic sandboxes:

| Sandbox | Weight | Scenario Type |
|---------|--------|---------------|
| **Document QA** | 30% | RAG scenarios — answer only from provided documents, cite sources, refuse out-of-scope |
| **Compliance Audit** | 25% | HIPAA/GDPR/SOX/CCPA — role-based access control, regulation-cited refusals |
| **Investigation** | 15% | Fraud/audit investigation — evidence provenance, conflicting testimony |
| **Rumor Propagation** | 15% | Information distortion — distinguish fact from hearsay |
| **Inquiry Learning** | 15% | Discovery-based education — knowledge boundaries |

## SFT Format

Each SFT example contains:
```json
{
"system": "You are DIEGETIC, an epistemically-constrained language model...",
"prompt": "<TASK>...</TASK>\n<OBS>...</OBS>\n<BELIEF>...</BELIEF>\n<MEM>...</MEM>\nUser query: ...\n<OUTPUT_JSON>",
"response": "{\"type\": \"diegetic_response\", \"utterance\": \"...\", \"epistemic\": {...}, \"action\": {...}}",
"metadata": {...}
}
```

## DPO Format

Each DPO pair contains:
- **chosen**: Epistemically correct response (grounded, cited, appropriately uncertain)
- **rejected**: Bad response (hallucinated, leaked information, overclaimed)

## Enterprise Roles

Training examples use these roles:
- `rag_assistant` — document-grounded QA
- `compliance_agent` — regulatory boundary enforcement
- `audit_investigator` — evidence-based investigation
- `tutor` — pedagogical knowledge boundaries
- `operator` — system monitoring with inference/observation distinction

## Usage

```python
from datasets import load_dataset

# Load SFT data
sft = load_dataset("howellx/diegetic-enterprise-training-data", data_files="enterprise_v1_sft.jsonl")

# Load DPO pairs
dpo = load_dataset("howellx/diegetic-enterprise-training-data", data_files="enterprise_v1_dpo.jsonl")
```

## Framework

Generated by the [DIEGETIC framework](https://github.com/justinrhowell/diegetic) — an open-source epistemic AI framework for building trustworthy, grounded AI systems.

## License

MIT