haizelabs/calligraphy-bench
收藏资源简介：
--- pretty_name: "Calligraphy Bench: Can AI Write Chinese Calligraphy?" language: - zh - en license: apache-2.0 size_categories: - n<1K task_categories: - image-to-image tags: - calligraphy - benchmark - evaluation - agent - tldraw - chinese - art dataset_info: config_name: default splits: - name: test --- # Calligraphy Bench **Can AI agents draw Chinese calligraphy stroke-by-stroke?** 88 of the hardest Chinese characters from 20 master calligraphers, drawn by 4 frontier AI agents on a tldraw canvas using programmatic stroke commands. Each output is evaluated on stroke correctness, stroke ordering, visual fidelity, and pairwise preference ranking. ## Task Given a reference calligraphy image (provided as a vision message — no file access), an AI agent must: 1. Draw the character stroke-by-stroke via MCP tools (`create_stroke`, `update_stroke`, etc.) 2. Control position, pressure (thick-thin variation), and size for each stroke 3. Capture and submit the final output via `save_document` No tracing, no bitmap copying, no image generation models — only programmatic strokes on a tldraw canvas through MCP tools. ## Models Evaluated | Model | Stroke Recall | Stroke Order | Fidelity | Pairwise Win Rate | |-------|:---:|:---:|:---:|:---:| | GPT-5.4 | 86% | 82% | 2.2/5 | **75%** | | Claude Opus 4.6 | 89% | 77% | 1.6/5 | 60% | | Kimi K2.5 | 83% | 79% | 1.3/5 | 34% | | GLM-5 | 71% | 78% | 1.3/5 | 30% | ## Evaluation Metrics - **Stroke Recall**: What fraction of reference strokes does the agent reproduce? (vision judge) - **Stroke Order**: Of matched strokes, what fraction are drawn in correct canonical order? (vision judge) - **Fidelity Verdict**: 1-5 scale comparing visual similarity to reference (vision judge) - **Pairwise Win Rate**: Head-to-head comparison using MJ1 judge (Tinker, round-robin all pairs) ## Data Fields - `task_id`: Unique identifier for the character+calligrapher combination - `character`: The Chinese character - `pinyin`: Romanized pronunciation - `definition`: English meaning - `stroke_count`: Number of canonical strokes - `style_difficulty`: 1-5 difficulty rating - `calligrapher`: Name of the master calligrapher - `reference_image`: Original calligraphy by the master calligrapher Each row is one character (88 total). This dataset contains only task inputs — model outputs and evaluation results are available on the [benchmark website](https://calligraphybench.com). ## Usage ```python from datasets import load_dataset ds = load_dataset("haizelabs/calligraphy-bench", split="test") print(f"{len(ds)} characters") # Browse tasks for row in ds: print(f"{row['character']} ({row['pinyin']}) — {row['calligrapher']}, {row['stroke_count']} strokes") ``` ## Harness Agents are run via [OpenHands SDK](https://github.com/All-Hands-AI/OpenHands) with MCP tools on a tldraw canvas. To submit your model for evaluation, contact [Haize Labs](https://haizelabs.com). ## Citation ```bibtex @misc{calligraphy-bench-2026, title={Calligraphy Bench: Can AI Write Chinese Calligraphy?}, year={2026}, url={https://huggingface.co/datasets/haizelabs/calligraphy-bench} } ```
