PipeBench : End to End ML Pipeline Benchmarking

This repository provides three large-scale synthetic datasets used in the manuscript associated with this record. The datasets were generated to evaluate the performance, scalability, and resource utilization of distributed machine learning pipelines and AutoML workflows in heterogeneous computing environments. The datasets simulate realistic characteristics from three important domains: Industrial Internet of Things (IoT) E-commerce user behavior and purchase conversion Financial market time-series and asset analytics All datasets are generated using a configurable synthetic data generation framework implemented in Python. The framework enables reproducibility, scalability, and controlled statistical properties suitable for benchmarking distributed machine learning systems. Dataset Components 1. IoT-Synth: Industrial Sensor Dataset This dataset simulates large-scale industrial IoT sensor streams commonly found in manufacturing and industrial monitoring systems. Key Characteristics Total samples: 100,000,000 Number of sensors: 15,000 Features: 128 numerical sensor features Classes: 5 failure modes (including normal operation) Failure rate: 3.2% Sampling frequency: 1 Hz equivalent File format: Parquet (Snappy compression) Feature Types The dataset includes: Synthetic sensor measurements (pressure, temperature, vibration) Correlated multivariate signals Sensor identifiers Timestamps Failure mode labels The features are generated using multivariate Gaussian distributions with controlled correlation structures, simulating realistic dependencies between sensors. Each failure mode modifies the mean and variance of sensor signals to emulate abnormal behavior. Example Fields sensor_0 ... sensor_127 sensor_id timestamp failure_mode This dataset is suitable for: Predictive maintenance Fault detection Time-series anomaly detection Large-scale AutoML benchmarking 2. Ecomm-Synth: E-Commerce Behavioral Dataset This dataset simulates user interaction and purchase behavior on an e-commerce platform. Key Characteristics Total samples: 15,000,000 sessions Features: 67 behavioral and demographic features Target variable: purchase conversion Session-level timestamps File format: Parquet (Snappy compression) Feature Categories The dataset includes several categories of variables: User demographics age gender income Behavioral metrics number of visits time spent on site pages viewed cart additions previous purchases Product attributes product category price Interaction indicators clicked recommendation used search viewed reviews The conversion label is generated through a logistic probability model combining user behavior and product characteristics, mimicking realistic online shopping dynamics. Example Fields age gender income n_visits time_on_site product_category price conversion session_id timestamp This dataset supports research in: recommendation systems user behavior modeling marketing analytics conversion prediction 3. Finance-Synth: Financial Market Dataset The finance dataset simulates high-frequency trading and market analytics scenarios. Key Characteristics Total samples: 5,000,000 observations Assets: 500 financial assets Features: 89 financial indicators Temporal frequency: 1-minute intervals File format: Parquet (Snappy compression) Data Generation Model The dataset uses a factor-based stochastic model that incorporates: common market factors asset-specific noise volatility clustering autocorrelated returns This produces time-series with statistical properties similar to real financial markets. Example Fields asset_id timestamp price returns volume order_imbalance rsi interest_rate vix forward_return_30min This dataset is useful for: algorithmic trading research financial forecasting time-series machine learning risk modeling