DJLougen/wittgensite

--- license: cc-by-nc-sa-4.0 task_categories: - text-generation language: - en tags: - benchmark - code-generation - prompt-consistency - ai-agents - coding-agents - evaluation - wittgenstein - semantic-invariance - caduceus pretty_name: "WittgenSite: Prompt Consistency Benchmark" size_categories: - n<1K configs: - config_name: prompts data_files: - split: test path: prompts.jsonl --- # WittgenSite: A Prompt Consistency Benchmark for AI Coding Agents **Created by [Daniel Lougen](https://huggingface.co/DJLougen)** Most benchmarks test whether an agent *can* complete a task. WittgenSite tests whether an agent produces **the same output regardless of how you ask**. Inspired by Wittgenstein's insight that meaning is determined by use — this benchmark measures whether AI coding agents extract the same meaning from semantically equivalent prompts. ## Benchmark Design One locked specification. 100 semantically different prompts. Same task every time. Score = consistency across runs. The agent builds a 5-page SaaS website from `GOLDEN-SPEC.md` (vanilla HTML + Tailwind CDN). Each run uses a different prompt from `PROMPTS.md`. The outputs are compared for structural, textual, behavioral, and stylistic consistency. ## Prompt Categories | Category | Prompts | Tests | |----------|---------|-------| | Direct & Minimal | 1-25 | Baseline consistency with simple instructions | | Role / Persona Based | 26-50 | Whether persona framing ("you are a senior dev") causes drift | | Verbose / Detailed | 51-75 | Whether extra detail causes additions or changes | | Casual & Red Herring | 76-100 | Whether suggestive language ("make it premium") causes deviation | ## Scoring ### Per-Run: Spec Fidelity (7 dimensions) | Dimension | Weight | |-----------|--------| | Structure & Files | 20% | | Copy Fidelity | 15% | | Theme System | 15% | | Accessibility | 15% | | Responsive Layout | 10% | | Interactivity | 15% | | Code Quality | 10% | ### Cross-Run: Consistency (5 dimensions) | Dimension | Weight | |-----------|--------| | Structural Consistency | 30% | | Copy Consistency | 25% | | Behavioral Consistency | 20% | | Style Consistency | 15% | | Exact Match Rate | 10% | ### Interpretation | Score | Meaning | |-------|---------| | 90-100 | Excellent — near-deterministic output | | 75-89 | Good — minor cosmetic drift | | 50-74 | Moderate — prompt wording affects output | | < 50 | Poor — output depends heavily on phrasing | ## Preliminary Results 2 runs tested (Prompt #1 direct vs. Prompt #26 role-based), same model: - Per-run spec fidelity: **100/100** and **99.5/100** - Cross-run consistency: **31.9/100 (Poor)** Both runs built functional websites that matched the spec individually. But the implementations were structurally different — different Tailwind classes, different JS patterns, different HTML nesting. Adding "You are a senior frontend developer" to the prompt changed the output significantly. ## Statistical Power Analysis How many runs do you need for reliable results? A formal power analysis (assuming SD ≈ 15 on the 0–100 consistency scale, based on pilot data): ### Estimating Overall Consistency | Desired Precision (95% CI) | Runs Needed | |----------------------------|-------------| | ±10 points | 9 | | ±7 points | 18 | | ±5 points | 35 | | ±3 points | 97 | ### Detecting Category Effects (ANOVA) Can prompt style (direct vs. persona vs. verbose vs. casual) cause consistency drift? | Runs per Category | Power (medium effect) | Power (large effect) | |-------------------|----------------------|---------------------| | 10 | 21% | 50% | | 15 | 32% | 71% | | 20 | 42% | 85% | | 25 (all) | 52% | 92% | ### Comparing Two Models | Runs per Model | Power (medium effect, d=0.5) | Power (large effect, d=0.8) | |----------------|------------------------------|----------------------------| | 20 | 34% | 69% | | 30 | 48% | 86% | | 50 | 70% | 98% | ### Recommended Run Counts | Goal | Minimum | Recommended | |------|---------|-------------| | Quick estimate of overall consistency | 9 runs (random sample) | 35 runs | | Per-category breakdown | 10 per category (40 total) | All 25 per category (100 total) | | Model-vs-model comparison | 30 runs each | 50 runs each | | Full benchmark (leaderboard submission) | All 100 prompts | All 100 prompts | > **Note:** Pilot data (2 runs, consistency = 31.9/100) suggests high variance across implementations. With only 25 prompts per category, **running all 100 is strongly recommended** for reliable per-category claims. ## Submission Requirements To submit results to the [WittgenSite Leaderboard](https://huggingface.co/spaces/DJLougen/Wittgensite-leaderboard), your submission must include: ### Required - **All 100 prompts executed** — partial runs are not accepted for leaderboard ranking - **Fresh context per run** — each prompt must start a new agent session with no memory of prior runs - **5 HTML output files per run** saved to numbered directories (`runs/001/` through `runs/100/`) - **Model identification** — exact model name, version, and provider (e.g., `claude-sonnet-4-20250514`, `gpt-4o-2024-08-06`) - **Agent framework** — tool/framework used (e.g., Claude Code, Cursor, Aider, custom harness) - **Scoring outputs** — results from both `scoring/evaluate.py` and `scoring/consistency.py` - **Temperature setting** — must be reported; default/recommended is the agent's default ### Submission Format ``` submission/ ├── meta.json # Model, agent, temperature, date, submitter ├── runs/ │ ├── 001/ # Prompt 1 output │ │ ├── home.html │ │ ├── features.html │ │ ├── pricing.html │ │ ├── about.html │ │ └── app.html │ ├── 002/ # Prompt 2 output │ │ └── ... │ └── 100/ │ └── ... ├── scores/ │ ├── fidelity.json # Per-run spec fidelity scores │ └── consistency.json # Cross-run consistency scores └── logs/ # Optional: raw agent logs for reproducibility ``` ### meta.json Example ```json { "model": "claude-sonnet-4-20250514", "agent": "Claude Code v1.0.0", "temperature": "default", "date": "2026-04-10", "submitter": "DJLougen", "notes": "Run on macOS, no custom system prompt" } ``` ### Disqualification Criteria - Runs sharing context or memory between prompts - Manual editing of output files - Cherry-picking a subset of prompts - Using the scoring scripts to iteratively improve outputs ## Limitations - **Subsampling is insufficient for category-level conclusions.** With only 25 prompts per category and expected high variance in consistency scores (SD ≈ 15), running fewer than all 25 prompts in a category does not provide enough statistical power to make reliable claims about whether a specific prompt style (e.g., persona-based vs. direct) causes meaningful drift. Full 100-prompt runs are required for per-category analysis. - **Pilot data is limited.** Current power estimates are based on 2 pilot runs. As more submissions arrive, the assumed variance (and therefore sample size recommendations) may be revised. - **Single-spec benchmark.** Results reflect consistency on one specific website specification. Generalization to other tasks or domains is not established. - **Scoring is automated, not human-judged.** The fidelity and consistency scorers use structural/textual comparison, which may miss semantic equivalences (e.g., two visually identical implementations using different CSS approaches). ## Files | File | Description | |------|-------------| | `GOLDEN-SPEC.md` | Locked website specification (source of truth) | | `PROMPTS.md` | 100 semantically diverse prompts across 4 categories | | `scoring/evaluate.py` | Per-run spec fidelity scorer | | `scoring/consistency.py` | Cross-run consistency scorer (primary metric) | | `task.json` | Caduceus benchmark metadata | ## Usage ```bash # 1. Run agent with fresh context + golden spec + one prompt # 2. Save output to runs/001/, runs/002/, etc. # Score individual run python scoring/evaluate.py runs/001/ # Score consistency across all runs python scoring/consistency.py runs/ ``` ## Citation ```bibtex @misc{lougen2026wittgensite, title={WittgenSite: A Prompt Consistency Benchmark for AI Coding Agents}, author={Lougen, Daniel}, year={2026}, url={https://github.com/DJLougen/wittgensite}, note={Inspired by Wittgenstein's philosophy of language} } ``` ## License **CC BY-NC-SA 4.0** — Attribution required, non-commercial, share-alike. Commercial use requires explicit written permission from [Daniel Lougen](https://x.com/DJLougen). ## Links - **Leaderboard**: [DJLougen/Wittgensite-leaderboard](https://huggingface.co/spaces/DJLougen/Wittgensite-leaderboard) - **GitHub**: [DJLougen/wittgensite](https://github.com/DJLougen/wittgensite) - **Caduceus Task Page**: [djlougen.github.io/caduceus/tasks/T014](https://djlougen.github.io/caduceus/tasks/T014) - **Author**: [@DJLougen](https://x.com/DJLougen)许可证：CC BY-NC-SA 4.0 任务类别： - 文本生成 语言： - 英语 标签： - 基准测试(Benchmark) - 代码生成(Code Generation) - 提示一致性(Prompt Consistency) - AI智能体(AI Agent) - 编码智能体(Coding Agent) - 评估 - 维特根斯坦(Wittgenstein) - 语义不变性(Semantic Invariance) - Caduceus 易读名称："WittgenSite：提示一致性基准测试集" 样本规模： - 少于1000条 配置项： - 配置名称：prompts 数据文件： - 拆分方式：测试集 - 路径：prompts.jsonl # WittgenSite：面向AI编码智能体的提示一致性基准测试集 **创作者：[Daniel Lougen](https://huggingface.co/DJLougen)** 多数基准测试仅评估智能体能否完成某项任务，而WittgenSite旨在检验智能体是否能在不同提问方式下生成完全一致的输出。 本基准测试受维特根斯坦(Wittgenstein)“意义即使用”的哲学洞见启发，旨在衡量AI编码智能体能否从语义等价的不同提示词中提取出完全一致的任务含义。 ## 基准测试设计 采用一份固定不变的任务规范，搭配100条语义各异的提示词，每次执行的任务完全一致，最终得分即为多次运行结果的一致性程度。 智能体需根据`GOLDEN-SPEC.md`中的规范搭建一个包含5个页面的SaaS网站（采用原生HTML + Tailwind CDN实现），每次运行使用`PROMPTS.md`中的一条不同提示词，最终通过比对各次运行的输出结果，评估其结构、文本、行为与风格层面的一致性。 ## 提示词类别 | 类别 | 提示词范围 | 测试目标 | |----------|---------|-------| | 直接极简型 | 1-25 | 评估简单指令下的基线一致性 | | 角色/人设导向型 | 26-50 | 检验人设框架（如“你是资深开发工程师”）是否会导致输出偏移 | | 详尽冗长型 | 51-75 | 检验额外细节是否会引发输出增改或偏离 | | 随意干扰型 | 76-100 | 检验暗示性语言（如“打造高端版本”）是否会导致输出偏差 | ## 评分规则 ### 单运行得分：规范契合度（7个维度） | 维度 | 权重 | |-----------|--------| | 结构与文件组织 | 20% | | 文本保真度 | 15% | | 主题系统 | 15% | | 可访问性 | 15% | | 响应式布局 | 10% | | 交互性 | 15% | | 代码质量 | 10% | ### 跨运行得分：一致性（5个维度） | 维度 | 权重 | |-----------|--------| | 结构一致性 | 30% | | 文本一致性 | 25% | | 行为一致性 | 20% | | 风格一致性 | 15% | | 精确匹配率 | 10% | ### 得分释义 | 得分区间 | 含义 | |-------|---------| | 90-100 | 优秀 — 输出近乎确定 | | 75-89 | 良好 — 仅存在细微外观偏移 | | 50-74 | 中等 — 提示词措辞会影响输出 | | < 50 | 较差 — 输出高度依赖措辞 | ## 初步测试结果 使用同一模型，分别测试了提示词1（直接型）与提示词26（角色导向型）： - 单运行规范契合度：分别为**100/100**与**99.5/100** - 跨运行一致性：**31.9/100（较差）** 两次运行均生成了符合规范的可用网站，但具体实现存在结构性差异：使用了不同的Tailwind类、不同的JavaScript实现模式与不同的HTML嵌套结构。仅在提示词中加入“你是资深前端开发工程师”就显著改变了最终输出。 ## 统计效力分析 要获得可靠的测试结果，需要执行多少次运行？基于预实验数据，本分析假设0-100分的一致性得分的标准差约为15，开展了正式的统计效力计算： ### 总体一致性估计 | 期望精度（95%置信区间） | 所需运行次数 | |----------------------------|-------------| | ±10分 | 9次 | | ±7分 | 18次 | | ±5分 | 35次 | | ±3分 | 97次 | ### 类别效应检测（方差分析） 提示词风格（直接型、人设型、冗长型、随意型）是否会引发一致性偏移？ | 每类别运行次数 | 中等效应效力 | 大效应效力 | |-------------------|----------------------|---------------------| | 10次 | 21% | 50% | | 15次 | 32% | 71% | | 20次 | 42% | 85% | | 25次（全类别） | 52% | 92% | ### 双模型对比 | 每模型运行次数 | 中等效应效力（d=0.5） | 大效应效力（d=0.8） | |----------------|------------------------------|----------------------------| | 20次 | 34% | 69% | | 30次 | 48% | 86% | | 50次 | 70% | 98% | ### 推荐运行次数 | 测试目标 | 最低要求 | 推荐值 | |------|---------|-------------| | 快速估计总体一致性 | 9次（随机抽样） | 35次 | | 按类别细分分析 | 每类别10次（总计40次） | 每类别25次（总计100次） | | 模型间对比 | 每模型30次 | 每模型50次 | | 完整基准测试（排行榜提交） | 全部100条提示词 | 全部100条提示词 | > **注：** 预实验数据（2次运行，一致性得分为31.9/100）显示不同实现间存在较高方差。由于每个类别仅包含25条提示词，**若要获得可靠的类别层面结论，强烈建议执行全部100条提示词的测试**。 ## 提交要求 若要向[WittgenSite排行榜](https://huggingface.co/spaces/DJLougen/Wittgensite-leaderboard)提交结果，提交内容需包含： ### 必填项 - **执行全部100条提示词** — 排行榜排名不接受部分运行的提交结果 - **每次运行使用全新上下文** — 每条提示词需开启独立的智能体会话，不得保留过往运行的记忆 - **每次运行生成5个HTML输出文件** — 保存至编号目录（`runs/001/`至`runs/100/`） - **模型标识** — 需明确模型全称、版本与服务商（例如`claude-sonnet-4-20250514`、`gpt-4o-2024-08-06`） - **智能体框架** — 需说明使用的工具/框架（例如Claude Code、Cursor、Aider或自定义测试框架） - **评分结果** — 需提供`scoring/evaluate.py`与`scoring/consistency.py`的运行结果 - **温度参数设置** — 必须上报，默认/推荐值为智能体的默认参数 ### 提交目录格式 submission/ ├── meta.json # 模型、智能体框架、温度参数、日期、提交者信息 ├── runs/ │ ├── 001/ # 提示词1的输出结果 │ │ ├── home.html │ │ ├── features.html │ │ ├── pricing.html │ │ ├── about.html │ │ └── app.html │ ├── 002/ # 提示词2的输出结果 │ │ └── ... │ └── 100/ │ └── ... ├── scores/ │ ├── fidelity.json # 单运行规范契合度得分 │ └── consistency.json # 跨运行一致性得分 └── logs/ # 可选：原始智能体运行日志，用于复现结果 ### meta.json 示例 json { "model": "claude-sonnet-4-20250514", "agent": "Claude Code v1.0.0", "temperature": "default", "date": "2026-04-10", "submitter": "DJLougen", "notes": "在macOS上运行，未使用自定义系统提示词" } ### 不合格提交规则 - 不同提示词的运行共享上下文或记忆 - 手动编辑输出文件 - 选择性选取部分提示词进行测试 - 使用评分脚本迭代优化输出结果 ## 局限性 - **子采样不足以支撑类别层面的结论**。由于每个类别仅包含25条提示词，且一致性得分预计存在较高方差（标准差≈15），若仅执行一个类别中少于25条的提示词，将无法获得足够的统计效力来可靠判断特定提示词风格（如人设型vs直接型）是否会引发显著偏移。若要开展类别层面的分析，需执行全部100条提示词的测试。 - **预实验数据有限**。当前的效力估计仅基于2次预实验运行。随着更多提交结果的积累，假设的方差（以及相应的样本量建议）可能会被修正。 - **单规范基准测试**。测试结果仅反映在特定网站规范下的一致性，尚未验证能否推广至其他任务或领域。 - **评分采用自动化方式，而非人工评判**。契合度与一致性评分基于结构/文本比对，可能会忽略语义等价的情况（例如两个视觉效果一致但采用不同CSS实现方案的产物）。 ## 文件说明 | 文件 | 说明 | |------|-------------| | `GOLDEN-SPEC.md` | 固定的网站任务规范（唯一权威来源） | | `PROMPTS.md` | 涵盖4个类别的100条语义多样的提示词 | | `scoring/evaluate.py` | 单运行规范契合度评分脚本 | | `scoring/consistency.py` | 跨运行一致性评分脚本（核心指标） | | `task.json` | Caduceus基准测试元数据 | ## 使用方法 bash # 1. 使用全新上下文、固定规范与单条提示词运行智能体 # 2. 将输出结果保存至 runs/001/、runs/002/ 等目录 # 评分单次运行结果 python scoring/evaluate.py runs/001/ # 对所有运行结果进行一致性评分 python scoring/consistency.py runs/ ## 引用格式 bibtex @misc{lougen2026wittgensite, title={WittgenSite: A Prompt Consistency Benchmark for AI Coding Agents}, author={Lougen, Daniel}, year={2026}, url={https://github.com/DJLougen/wittgensite}, note={Inspired by Wittgenstein's philosophy of language} } ## 许可证 **CC BY-NC-SA 4.0** — 需注明原作者，非商业用途，相同方式共享。商业使用需获得[Daniel Lougen](https://x.com/DJLougen)的明确书面许可。 ## 相关链接 - **排行榜**：[DJLougen/Wittgensite-leaderboard](https://huggingface.co/spaces/DJLougen/Wittgensite-leaderboard) - **GitHub仓库**：[DJLougen/wittgensite](https://github.com/DJLougen/wittgensite) - **Caduceus任务页面**：[djlougen.github.io/caduceus/tasks/T014](https://djlougen.github.io/caduceus/tasks/T014) - **作者**：[@DJLougen](https://x.com/DJLougen)