GitLab Pipeline Resource Usage Dataset for Best Practice Comparison: System Monitoring, GitLab API Metadata, and CPU-Power Mapping

This dataset provides resource usage metrics from GitLab CI/CD pipelines, collected to facilitate a comparative analysis of different optimization strategies. The data is organized into scenarios, each representing a specific pipeline configuration: a baseline (base), dependency caching (caching), parallel job execution (parallel), and a combination of both (caching-parallel). Each scenario directory contains four CSV files: pipeline_meta.csv and job_meta.csv hold metadata retrieved from the GitLab API, including pipeline and job IDs, status, and precise start and end timestamps. pipeline_sar.csv contains system-wide resource metrics from the runner host, captured for the entire duration of a pipeline run. jobs_sar.csv offers resource data collected within the execution environment of individual jobs. Due to the design of the GitLab Docker executor, this file captures metrics exclusively from the main job execution step and does not include the pre-job (e.g., cloning, fetching caches) or post-job (e.g., creating caches) step, which run in separate helper containers. The dataset also includes server_idle_sar_1h.csv, which provides an hour of baseline resource measurements from the idle server, and intel-xeon-6246_vhr_167_cpu-watt-mapping.sh, a script related to the power estimation model (see below). Methodology Data collection was performed within a controlled environment using an automated process to ensure reproducibility. A reference pipeline, composed of build, test, and deploy stages, served as the foundation for all scenarios. The different best-practice scenarios were implemented as follows: Dependency Caching: To isolate the effects of caching, pipeline runs were conducted in pairs. A shared cache key was assigned to two consecutive pipeline executions, ensuring that the first run in each pair builds and populates the cache while the second run utilizes it. This design means that measurements from odd-numbered iterations correspond to cache initialization, whereas even-numbered iterations reflect cache usage. Furthermore, the configuration ensures that the build-api and test-api jobs access the same dependency cache within a single pipeline, since they use largely the same dependencies. Parallelization: Parallel execution of jobs was enabled by adjusting the GitLab Runner configuration to allow for multiple concurrent jobs (concurrent = 8) and by defining explicit job dependencies with the needs keyword in the .gitlab-ci.yml file. This setup allows independent jobs, such as the two build jobs, to run simultaneously. Combined Practices: A third scenario combines both caching and parallelization to measure their synergistic effects. Data Collection Environment All experiments were conducted on a dedicated GitLab Runner host to ensure consistent and comparable results. The host was a virtual machine provisioned with 8 vCPUs and 16GB of RAM, running on Ubuntu 24.04.3 LTS. The underlying physical server was equipped with two Intel Xeon Gold 6246 processors (Cascade Lake, 2019) and 384GB of RAM. The software stack included GitLab Runner version 18.1.1 configured with the Docker executor (version 28.3.0). Resource metrics were gathered using the System Activity Reporter (sar), which recorded CPU utilization, memory usage, I/O activity and other metrics at one-second intervals. Measurements were taken at two levels: on the host machine to capture the overall resource footprint of a pipeline, and inside the individual job containers to analyze job-specific consumption. Power Estimation The power consumption mapping included in the dataset are estimates. These were calculated using the Cloud Energy Model, an open-source project from the Green Software Foundation. The model estimates power draw primarily based on CPU utilization metrics, calibrated with hardware-specific parameters of the test environment.