eromang/eu-cyber-llm-benchmark-prompts

---
license: mit
task_categories:
- text-generation
language:
- en
tags:
- cybersecurity
- geopolitical-bias
- benchmark
- threat-assessment
- eu
- evaluation
pretty_name: EU Cyber Threat Landscape LLM Benchmark — Prompts
size_categories:
- n<1K
---

# EU Cyber Threat Landscape LLM Benchmark — Prompts

A research-grade evaluation benchmark for measuring geopolitical bias in LLM-generated cyber threat landscape assessments.

## What this is

A set of structured prompts designed to test whether language models exhibit actor-asymmetric framing when generating strategic cyber threat assessments in EU contexts. Each prompt describes a cyber incident in a specific critical infrastructure sector, paired with an attribution condition that varies the suspected or confirmed threat actor.

The incident description and sector scope are held constant within each scenario. Only the attribution framing changes. This isolates the effect of geopolitical framing on model output.

## Splits

| Split | Scenarios | Conditions | Prompts | Actors |
|-------|-----------|------------|---------|--------|
| `phase_1` | 20 | 5 | 200 | Neutral, China, Russia (Suspected/Confirmed) |
| `phase_2` | 48 | 11 | 528 | Neutral, China, Russia, US, Iran, DPRK (Suspected/Confirmed) |

Phase 2 expands Phase 1 with 28 additional scenarios across 7 thematic blocks (EU internal, Chinese tech, multipolar, false-flag, non-state, democratic process, vendor-specific) and 6 additional conditions (US, Iran, DPRK attribution).

## Schema

| Field | Type | Description |
|-------|------|-------------|
| `prompt_id` | string | Unique identifier (e.g., `S21_China_Confirmed`) |
| `scenario_id` | string | Scenario group (e.g., `S21`) |
| `condition` | string | Attribution condition (e.g., `China_Confirmed`, `Neutral`) |
| `sector_focus` | string | Critical infrastructure sector (e.g., `Energy`, `Health`) |
| `prompt_text` | string | Full prompt text |

## How to use

```python
from datasets import load_dataset

ds = load_dataset("eromang/eu-cyber-llm-benchmark-prompts")

# Phase 2 prompts
for row in ds["phase_2"]:
print(row["prompt_id"], row["condition"], row["sector_focus"])
```

### Run against a local model (Ollama)

```python
import requests
from datasets import load_dataset

ds = load_dataset("eromang/eu-cyber-llm-benchmark-prompts", split="phase_2")

for row in ds:
resp = requests.post("http://localhost:11434/api/generate", json={
"model": "llama3.1:8b-instruct-q4_K_M",
"prompt": row["prompt_text"],
"stream": False,
"options": {"temperature": 0.0, "num_ctx": 4096},
})
print(row["prompt_id"], len(resp.json()["response"]))
```

## Controlled variables

- Incident description is constant within each scenario
- Sector scope is constant within each scenario
- Only the attribution framing varies between conditions
- Analytical instructions are identical across all prompts
- Operational detail is prohibited in prompt templates

## Sectors covered

Energy, Health, Transport, Finance, Digital Infrastructure, Water, Space, Defence, Telecommunications, Government, Maritime, Supply Chain, and others across EU critical infrastructure.

## Citation

```bibtex
@misc{romang2026eucyberbenchmark,
author = {Eric Romang},
title = {EU Cyber Threat Landscape LLM Benchmark: Geopolitical Bias in Local Language Models},
year = {2026},
url = {https://github.com/eromang/researches/tree/main/LLM-Benchmark},
note = {Research benchmark for evaluating actor-asymmetric framing in local LLMs}
}
```

## Related

- [Response dataset](https://huggingface.co/datasets/eromang/eu-cyber-llm-benchmark-responses) — 15,988 model responses across 7 models
- [GitHub repository](https://github.com/eromang/researches/tree/main/LLM-Benchmark) — Full analysis scripts, per-model reports, methodology

## License

MIT