Artifact: On the risk of coding before testing: An empirical study on LLM-based test generation workflow

Artifact: On the risk of coding before testing: An empiricalstudy on LLM-based test generation workflow This artifact contains the data accompanying our paper. It includes the benchmarks, prompts, faulty implementations, and generated test suites used throughout the empirical evaluation. Repository Structure ```.├── benchmarks/├── prompts/├── faults/└── tests/``` `benchmarks/` This directory contains the benchmark datasets used in our study. The original benchmark records are **unmodified**. * `HumanEval+/he.jsonl` — HumanEval+ benchmark* `MBPP/mbpp.jsonl` — MBPP benchmark* `BigCodeBench/bcb.jsonl` — BigCodeBench benchmark The directory also contains the `mbpp_*.jsonl` files, which correspond to the mutated MBPP prompts introduced by prior work. The only modification compared to the previous work's files is the renaming of certain key names to match the schema used by the other benchmark files, enabling a uniform processing pipeline. The rest of the benchmark contents themselves remain unchanged. Please note: `prompts/` This directory contains all prompts used during code generation and test generation. The implementation used to execute the experiments will be released upon paper acceptance. `faults/` This directory contains the faulty implementations collected during **Phase 1** of the methodology described in the paper. The directory is organized as follows: ```faults/└── <Benchmark>/ └── <Model>/ └── faults.jsonl``` Each `faults.jsonl` file contains the faulty implementations generated by the corresponding model for the specified benchmark. `tests/` This directory contains the generated unit test suites used in the evaluation. Test suites are organized according to the benchmark, model, and test-generation configuration reported in the paper. Each test suite corresponds to one of the collected faulty implementations and represents the output of a particular experimental configuration (e.g., prompt-only, code-only, prompt engineering techniques, or agentic workflow). A few notes on the naming convention:- original-minimal: Prompt only (Test-Driven)- none: Code only- from-complete-history: Agentic workflow Reproducing the Experiments This artifact contains all datasets produced during the study. The source code used to generate the faulty implementations, produce the test suites, and evaluate the results will be released upon paper acceptance.