figmatrace

# Dataset Card for FigmaTrace FigmaTrace is a dataset of expert human Figma design workflows: 200+ hours of screen-recorded work converted into 3,469 agent trajectories using a design phase-based segmentation method. It is built to teach vision language models the creative skills and decisions behind design work, not just the finished artifact. ## Dataset Details ### Dataset Description - **Curated by:** Patronus AI - **Language(s):** English - **License:** CC-BY-4.0 ### Dataset Sources - **Raw Assets:** [Google Drive](https://drive.google.com/drive/folders/1d7NQxjiAzALu3odUQSxbT6eO9czm96Oy?usp=drive_link) - **Paper:** [FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows](https://cdn.patronus.ai/FigmaTrace.pdf) - **Best fine-tuned model:** https://huggingface.co/PatronusAI/Qwen3.8-27B-Figmatrace-SFT ## Uses ### Direct Use Supervised fine-tuning and evaluation of VLM-based GUI/design agents. ### Out-of-Scope Use Trajectories carry no pre-annotated reasoning chains, so the dataset is not suited for training reasoning-trace models without further annotation. ## Dataset Structure - 126 long-horizon tasks across 8 designer workflow categories (pixel-perfect replication, responsive adaptation, theming with variables, sketch-to-Figma, flaw injection/repair, edge-content resilience, a11y remediation, prototype wiring), covering a 10-skill expert taxonomy. - 3,469 trajectories: 2,883 training / 586 evaluation. - Each trajectory is a sequence of action-frame pairs in the Playwright-MCP action space (e.g. `mouse_click`, `keyboard_type`), with `observe` probes inserted for input-free screen transitions. - Trajectories carry phase labels from a closed 12-label vocabulary (e.g. `blocking_layout`, `componentising`, `refinement_polish`) and skill labels assigned by frequency.  ## Dataset Creation ### Curation Rationale Existing design datasets capture final artifacts rather than the sequence of decisions that produced them. FigmaTrace records full expert sessions so agents can learn the workflow itself. ### Source Data #### Data Collection and Processing OS-level actions and screen captures were recorded from experts solving the 126 tasks. Processing: ~95% of raw actions (idle mouse movement) filtered out; two-pass frame extraction with settle detection; effect filtering by changed-pixel fraction; phase segmentation with Gemini-3.6-Flash using consensus boundaries across 3/6/12-way shardings. Total compaction: 179x versus raw OS events. #### Who are the source data producers? Subject-matter experts hired through Upwork, each with 2+ years of Figma experience, aged 18+, and vetted with a starter task. ## Bias, Risks, and Limitations - Open-ended tasks (theming, sketch-to-Figma, prototyping) reflect individual SME preferences by design. - Some noisy actions leak through preprocessing; models trained on the data can repeat near-identical clicks or over-favor screen-center targets. ## Citation **BibTeX:** ```bibtex @article{deshpande2026figmatrace, title={FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows}, author={Deshpande, Darshan and Fujinuma, Yoshinari and Markiewicz, Martyna and Bansal, Devanshu and Jain, Shivani and Saban, Nicholas and Maheshwari, Chirag and Kannappan, Anand}, journal={https://cdn.patronus.ai/FigmaTrace.pdf}, year={2026} } ``` Dataset Card Contact [e-mail]