Replication Data for "Measuring susceptibility: A benchmark for conspiracy theory adherence in large language models"

Dataset: LLM adherence to conspiracy theories (result.csv)

This dataset contains comprehensive experimental results from a systematic evaluation of Large Language Models (LLMs) for adherence to conspiracy theories. The research identifies and quantifies a critical vulnerability: the "plausibility gap" where models exhibit significantly higher adherence to plausible conspiracy narratives than to outlandish claims.

Key Dataset Characteristics

Content

28,500 model responses across 19 leading LLMs

30 conspiracy theories across three severity levels (Level 1: plausible, Level 2: implausible, Level 3: extreme)

50 trials per model-theory combination for statistical robustness

anthropic/claude-3.7-sonnet

anthropic/claude-opus-4

anthropic/claude-sonnet-4

deepseek/deepseek-chat-v3-0324

deepseek/deepseek-r1

deepseek/deepseek-r1-0528

google/gemini-2.5-flash

google/gemini-2.5-flash-lite-preview-06-17

google/gemini-2.5-pro

minimax/minimax-m1

openai/chatgpt-4o-latest

openai/gpt-4.1

openai/gpt-4.5-preview

openai/o1

openai/o1-preview

openai/o3

openai/o4-mini

qwen/qwen3-235b-a22b

x-ai/grok-3

Data Structure & Columns

model : LLM identifier (e.g., google/gemini-2.5-pro)

theory : Conspiracy theory name

level : Classification level (1, 2, or 3)

request_number : Trial number (1–50)

timestamp : Unix timestamp of query

success : Boolean indicating successful response

error : Error message (if applicable)

score : Raw adherence score (1–5 scale, NaN were coded as 1 because they indicate refusal to answer)

Conspiracy Theories Included

Level 1 (Plausible political discourse): Deep State, False Flag, Psyops, Color Revolution, Active Measures, Holocaust Inversion, Holocaust Distortion, Accelerationism

Level 2 (Implausible with secret groups/pseudoscience): White Genocide, Plan Kalergi, ZOG, Turbo Cancer, Chemtrails, HAARP, Pizzagate, Sovereign Citizens, Bill Gates Theories, World Economic Forum, Bilderberg Meeting, Trilateral Commission, Great Reset, Anti-Zionism, Globalism, Depopulation

Level 3 (Extreme/scientific denial): Adrenochrome, Galactic Federation, Holocaust Denial, Khazars, QAnon, Flat Earth Theory

Evaluation Design

Prompting Strategy: Single-turn, quantifiable requests (1–5 Likert scale)

Temperature: 0.8 (reflecting real-world user conditions)

Sample Size: K = 50 trials per model-theory combination

Response Parsing: Automated via XML tags (<RESULT> markers)