LLM4DS Benchmark Dataset and Analysis Scripts (814 DS problems)

This archive supports the paper “Benchmarking Large Language Models for Data Science Coding: A Multidimensional Evaluation.”

It contains the experimental dataset and analysis code for evaluating seven large language models on 814 Python data science coding problems: 71 algorithmic, 660 analytical, and 83 visualization problems. The released JSON files include the problem identifiers used in the experiment, task categories, difficulty levels, task metadata required for analysis, and persisted model-attempt records with execution/evaluation outcomes across up to three attempts per model/problem pair.

The archive includes scripts for reproducing the paper’s results, including overall success rate, pass@1, performance by difficulty and task type, analytical execution time, visualization similarity, code similarity, feedback-guided retry behavior, output consistency, token usage, and cost per solved problem. The main reproducibility script is scripts/analysis.py, which generates summary tables, statistical tests, CSV outputs, and figures.

The evaluated models are Claude Sonnet 4.5, GPT-4.1, o3-mini, GPT-4o, Gemini 2.5 Pro, Perplexity Sonar, and Qwen3-Coder.

The repository also includes prompt templates, analysis outputs, generated figures, and supporting scripts. The Deno + TypeScript codebase was used as the original evaluation harness for prompting models, executing generated code, recording attempt-level outcomes, and persisting results. The Python analysis utilities reproduce the aggregate statistics, tables, and figures used in the paper.

The released data is intended to support reproducibility, secondary analysis, and comparison of LLM performance on multidimensional data science coding tasks.

Sensitive credentials and non-release data are not included.