vibe-design-arena

# Vibe Design Arena The first public pairwise human preference dataset for real-world vibe-coded web applications. 60 apps from the [Vibe Coding Showcase](https://vibecodingshowcase.com/) were compared pairwise by human annotators who judged which app has better visual design based on screenshots. Every possible pair was evaluated (C(60,2) = 1,770 comparisons), with 30 human votes per pair. ## Dataset Summary | Stat | Value | |------|-------| | Apps | 60 | | Pairwise comparisons | 1,770 | | Human votes per comparison | 30 | | Total human judgments | ~53,000 | | Screenshot resolution | 1440×900 | ## Subsets ### `apps` (60 rows) One row per app with metadata and aggregate rankings. | Field | Type | Description | |-------|------|-------------| | `app_id` | string | Unique identifier (derived from screenshot filename) | | `app_name` | string | Project name | | `website` | string | Live URL | | `description` | string | Project description | | `tools` | list[string] | Normalized vibe coding tools (e.g. `["Bolt", "Cursor"]`) | | `frameworks` | list[string] | Frameworks and libraries used | | `build_time_raw` | string | Original build time text (e.g. "2 weeks") | | `build_time_days` | float | Normalized build time in days | | `revenue` | string | Revenue information (self-reported) | | `creator` | string | Creator name | | `location` | string | Creator location | | `screenshot` | image | 1440×900 screenshot of the app | | `wins` | int | Number of pairwise matchups won (out of 59) | | `losses` | int | Number of pairwise matchups lost | | `win_rate` | float | wins / (wins + losses) | | `rank` | int | 1–60, sorted by win rate | ### `comparisons` (1,770 rows) One row per pairwise comparison with vote counts and both screenshots. | Field | Type | Description | |-------|------|-------------| | `app_a` | string | App ID of first app | | `app_b` | string | App ID of second app | | `image_a` | image | Screenshot of app A | | `image_b` | image | Screenshot of app B | | `votes_a` | int | Number of votes for app A (out of 30) | | `votes_b` | int | Number of votes for app B (out of 30) | | `winner` | string | `"app_a"` or `"app_b"` | ## Usage ```python from datasets import load_dataset # Load app metadata and rankings apps = load_dataset("datapointai/vibe-design-arena", "apps") # Load pairwise comparisons comparisons = load_dataset("datapointai/vibe-design-arena", "comparisons") ``` ## Key Findings ### Tool Rankings (by aggregate win rate) 1. **Bolt** — 58.6% win rate (21 apps) 2. **Cursor** — 57.3% win rate (27 apps) 3. **Claude** — 51.3% win rate (16 apps) 4. **Lovable** — 45.1% win rate (13 apps) 5. **ChatGPT** — 42.2% win rate (13 apps) 6. **Replit** — 45.3% win rate (8 apps) ### Build Time vs Design Quality No meaningful correlation (Pearson r = 0.23). The #2 ranked app (ActivDNA, 95% win rate) was built in 3 days. ### Revenue Only 3 of 60 apps report actual revenue. No correlation between design quality and revenue. ## What Makes This Dataset Unique - **Real shipped products** — not AI-generated from synthetic prompts - **Complete pairwise matrix** — every app compared against every other app - **Rich metadata** — tools, frameworks, build time, revenue, creator info - **Large-scale annotation** — 30 votes per pair, ~53K total human judgments - **Vibe coding focused** — all apps built with AI-assisted development tools ## Comparison to Related Work | Dataset | Type | Apps | Pairs | Votes | Real Products? | |---------|------|:----:|:-----:|:-----:|:--------------:| | **Vibe Design Arena** | Pairwise preference | 60 | 1,770 | ~53K | Yes | | UI-Bench | Pairwise preference | 300 | — | 4K+ | No (AI-generated) | | DesignPref | Pairwise preference | — | 12K | 12K | No (generated) | | UICrit | Critique/rating | 983 | — | 3K | No (mobile UIs) | ## Data Collection Apps were sourced from the [Vibe Coding Showcase](https://vibecodingshowcase.com/) hackathon. Screenshots were captured programmatically at 1440×900 resolution using Playwright. Pairwise comparisons were conducted on [Datapoint AI](https://datapoint.ai/), where human annotators viewed two screenshots side-by-side and selected the one with better visual design. Each comparison received 30 independent votes. ## License CC-BY-4.0 ## Citation ```bibtex @dataset{vibe_design_arena_2025, title={Vibe Design Arena: A Pairwise Human Preference Dataset for Vibe-Coded Web App Design}, author={Datapoint AI}, year={2025}, url={https://huggingface.co/datasets/datapointai/vibe-design-arena} } ```