andremrincon/emse_covertest: v1.0

Replication package for the empirical study "Incremental Multi-Model LLM Strategy for Automated Open-Box Integration Testing of REST APIs" (submitted to Empirical Software Engineering). It contains the raw execution records, the generated test suites, the subject systems, and the analysis scripts needed to reproduce every number reported in the paper. The study evaluates a coverage-guided pipeline that starts from an EvoMaster baseline suite (T0) and applies three LLMs in sequence, each targeting the instructions left uncovered by the previous stage. The tool that produced these data, COVERTEST, is released separately at github.com/aurimrv/COVERTEST. Experimental design A full factorial design crossing four factors: Model ordering (6 levels): all permutations of qwen3.7-plus, gpt-5-mini and glm-5.2, accessed through OpenRouter Subject system (5 levels): features-service, ncs, restcountries, scs, spring-actuator-demo, all from the Web Fuzzing Dataset Random seed (3 levels): 367, 450, 767 Repetition (3 levels): three independent runs 6 × 5 × 3 × 3 = 270 executions, each comprising three generation stages, for 810 stages in total. Every execution is preserved in full: generated JUnit sources, compiled classes, JaCoCo reports, Surefire reports, LLM prompt/response traces and per-stage metrics. Campaign totals: US$ 131.16, 128,231,475 tokens, 10,740 API requests and 92.3 hours of pipeline execution, excluding the generation of the EvoMaster baselines. Contents analysis/ — extraction and analysis scripts (Python) plus the two curated datasets: executions_repaired.csv (270 rows, one per execution) and steps_repaired.csv (810 rows, one per generation stage) generated_tests/ — the raw data, one folder per execution, organized as sequence_NN / run_NN / seed / subject. The file ts01_synthesis_metrics.json is the execution record that the analysis reads; the remaining artifacts are supporting evidence projects/ — the five subject systems, each with its original and enriched OpenAPI specification, the runnable *-sut.jar, and the baseline JaCoCo report README.md — full documentation, including the mapping from data to research questions Mapping data to the research questions RQ1 (coverage improvement and its distribution across stages): steps[].initial_coverage, steps[].final_coverage and steps[].coverage_gain in each execution record. The comparison anchor for the multi-model claim is the coverage after the first LLM, not the EvoMaster baseline. RQ2 (effect of model ordering): the six sequence_NN folders are the ordering ablation. Comparing two sequences that share their first model isolates the effect of swapping the later positions. RQ3 (marginal cost–benefit profile): steps[].llm_totals and totals hold tokens, requests, and cost per stage and per cycle. Because token counts are recorded per stage, every monetary figure in the paper can be recomputed under any future price schedule. Reproducing the analysis pip install pandas numpy scipy matplotlibpython analysis/my_extract.py # raw JSON -> flat CSVpython analysis/final_analysis.py # recompute every reported quantitypython analysis/refigure.py # regenerate the figures All efficiency figures reported in the paper are computed as ratios of means (mean gain of a group divided by its mean cost), not as averages of per-execution ratios. The efficiency block inside the JSON records is a per-stage convenience field and is not what the paper reports. Data quality The dataset was verified before analysis for balance, stage count, continuity of the coverage chain, and reconciliation of the token and cost counters. Two findings are stated openly: Sixty of the 810 stages made no model call (zero tokens, zero cost, zero gain). All sixty belong to spring-actuator-demo at positions 2 or 3, in runs where the preceding stage had already reached 100% instruction coverage and the stop criterion was met. These are legitimate observations of a model with nothing left to contribute and are retained in every analysis. Two executions recorded an unattributed increment. In sequence_06/run_01/450/features-service and sequence_06/run_03/767/features-service, the cycle total is valid (56.02% → 80.96%), but all three stages report a zero gain. In the other seven runs of the same ordering and subject, the first stage gains exactly the same 24.94 pp, so the increment was reassigned to position 1. The raw JSON files are unmodified; the repair is applied in analysis/final_analysis.py. The paper reports a sensitivity analysis showing that excluding these two executions does not change the conclusion. Environment Java 11.0.27 · Maven 3.9.11 · EvoMaster 5.0.2 · JaCoCo 0.8.13 · JUnit 4 with REST Assured 4.5.1 · Python 3. Generation and repair used temperature = 0.2, top_p = 1.0, and a fixed maximum token budget per call, with gpt-5-mini held constant as the repair model across every configuration.