autocad-bench-tasks

# AutoCAD-Bench Tasks This dataset contains the **50 AutoCAD tasks used for model evaluation** in AutoCAD-Bench. ## Contents - 50 total evaluation tasks - 21 dimensioned 2D drafting tasks - 29 3D modeling and presentation tasks - millimetres as the task unit - one reference PNG and one scorer-side gold DWG per task - the exact 2D and 3D task prompts used by the benchmark Each task is stored under `data/task-NNN/`: ```text data/task-001/ ├── reference.png └── gold.dwg ``` `tasks.jsonl` records the sequential task ID, drawing type, difficulty level, units, task prompt, artifact paths, file sizes, and SHA-256 checksums. ## Evaluation boundary The model should receive only the task prompt and `reference.png`. The gold DWG is an evaluator-side artifact and must never be mounted in the model-controlled desktop, included in the model prompt, or otherwise made available to the agent during a rollout. For 2D tasks, the expected submission is a complete dimensioned drawing in Model Space. For 3D tasks, the expected submission contains genuine 3D geometry in Model Space and the complete reference-sheet presentation on `Layout1`. ## Loading the manifest ```python import json from pathlib import Path tasks = [ json.loads(line) for line in Path("tasks.jsonl").read_text().splitlines() if line.strip() ] ``` The DWG files use the AutoCAD 2018 `AC1032` file format.