ILTS-SINDy data

Benchmark Models with Noise Realizations Dataset Overview This dataset contains synthetic time series data generated from three canonical benchmark models (SIR, Lorenz, and Lotka-Volterra) with systematically introduced noise and outliers. The data is designed for testing and benchmarking inference, filtering, and state estimation algorithms under varying noise and contamination conditions. Dataset Composition The dataset consists of 3 files, one for each benchmark model: SIR_heatmap_100_data.npz – Susceptible-Infected-Recovered epidemic model LORENZ_heatmap_100_data.npz – Chaotic Lorenz attractor model LV_heatmap_100_data.npz – Predator-prey Lotka-Volterra model Each file contains 100 independent noise realizations for every combination of: Noise levels: 0.025, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2 Outlier percentages: 0.025, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2 (as fraction) Plus baseline: noise_level=0 with no outliers Total realizations per file: 100 × (8 noise levels + 1 baseline) × 8 outlier percentages Data Format File Format Format: NPZ (NumPy compressed binary format) Extension: .npz Compression: Included (ZIP compression) How to load: np.load('filename.npz', allow_pickle=True) Data Structure The NPZ file is organized hierarchically as a dictionary of dictionaries: ├── noise_0 │ ├── outInd_0: array of shape (100,) [baseline, no outliers] │ └── data_0: array of shape (100, 1001, n_states) ├── noise_0.025 │ ├── outInd_0.025: array of shape (100,) [outlier indices for each realization] │ ├── data_0.025: array of shape (100, 1001, n_states) │ ├── outInd_0.05: array of shape (100,) │ ├── data_0.05: array of shape (100, 1001, n_states) │ └── ... [for each outlier percentage] ├── noise_0.05 │ └── ... [similar structure for each noise level] └── ... [for remaining noise levels] Dimensions Realizations: 100 (independent samples per parameter combination) Time points: 1001 (equally spaced) States: 3 (SIR, Lorenz) or 2 (Lotka-Volterra) Loading and Accessing Data import numpy as np # Load the NPZ file data = np.load('LORENZ_heatmap_100_data.npz', allow_pickle=True) # Get a specific realization noise_level = 0.1 outlier_percent = 0.05 realization_id = 0 # Access the noisy data: shape (1001, n_states) noisy_data = data[f'noise_{noise_level:g}'][f'data_{outlier_percent:g}'][realization_id] # Access the outlier indices: shape (n_outliers,) outlier_indices = data[f'noise_{noise_level:g}'][f'outInd_{outlier_percent:g}'][realization_id] # The baseline clean data (noise_level=0, no outliers) clean_data = data['noise_0']['data_0'][0] # shape (1001, n_states) Model Descriptions 1. SIR Model (Susceptible-Infected-Recovered) File: SIR_heatmap_100_data.npz The standard epidemiological SIR compartmental model: dS/dt = -β·S·I dI/dt = β·S·I - γ·I dR/dt = γ·I Parameters: β (transmission rate) = 0.3 γ (recovery rate) = 0.1 States: Column 0: S(t) – Susceptible Column 1: I(t) – Infected Column 2: R(t) – Recovered Initial conditions: S₀=0.99, I₀=0.01, R₀=0 Integration interval: t ∈ [0, 100] Time points: 1001 equally spaced points 2. Lorenz Model File: LORENZ_heatmap_100_data.npz The classical chaotic Lorenz system: dx/dt = σ(y - x) dy/dt = x(ρ - z) - y dz/dt = xy - βz Parameters: σ = 10 (Prandtl number) ρ = 28 (Rayleigh number) β = 8/3 (geometric factor) States: Column 0: x(t) Column 1: y(t) Column 2: z(t) Initial conditions: x₀=-8, y₀=7, z₀=27 Integration interval: t ∈ [0, 20] Time points: 1001 equally spaced points 3. Lotka-Volterra Model File: LV_heatmap_100_data.npz The predator-prey Lotka-Volterra system: dx/dt = α·x - β·x·y (prey population) dy/dt = β·x·y - 2α·y (predator population) Parameters: α = 1.0 (prey growth rate) β = 0.1 (predation coefficient) States: Column 0: x(t) – Prey population Column 1: y(t) – Predator population Initial conditions: x₀=1.0, y₀=2.0 Integration interval: t ∈ [0, 30] Time points: 1001 equally spaced points Noise and Outlier Generation Noise Model Type: Additive Gaussian noise Application: observation = true_state + noise where noise ~ N(0, σ²) and σ = noise_level × RMS(state) Calculation: The noise standard deviation is scaled by the root mean square (RMS) of each state variable: σ_scaled = noise_level × √(mean(state²)) This ensures noise is relative to the magnitude of each state variable, making the noise levels comparable across different models and state scales. Outlier Percentages Type: Additive impulse noise at randomly selected time points Mechanism: Randomly select a fraction of time points based on outlier_percent For selected time points, inject Gaussian noise with standard deviation = noise_level × RMS(state) A boolean mask records which time points were corrupted Outlier percentages available: 2.5%, 5%, 7.5%, 10%, 12.5%, 15%, 17.5%, 20% Plus a baseline with 0% outliers Note: Outlier locations are different for each realization to simulate independent noise realizations. Usage Examples Python with NumPy import numpy as np import matplotlib.pyplot as plt # Load data data = np.load('LORENZ_heatmap_100_data.npz', allow_pickle=True) # Get a specific realization noise_level = 0.1 outlier_percent = 0.05 real_id = 0 noisy = data[f'noise_{noise_level}'][f'data_{outlier_percent}'][real_id] outliers = data[f'noise_{noise_level}'][f'outInd_{outlier_percent}'][real_id] clean = data['noise_0']['data_0'][0] # Plot: true state vs noisy observation time = np.linspace(0, 20, 1001) plt.figure(figsize=(12, 4)) plt.plot(time, clean[:, 0], 'b-', label='True state (x)', linewidth=1) plt.plot(time, noisy[:, 0], 'r.', label='Noisy observation', markersize=2) plt.scatter(time[outliers.flatten()], noisy[outliers.flatten(), 0], color='orange', s=30, label='Outliers', zorder=5) plt.xlabel('Time') plt.ylabel('State x') plt.legend() plt.title(f'Lorenz: noise_level={noise_level}, outlier_percent={outlier_percent*100:.1f}%') plt.show() Batch Processing All Realizations import numpy as np data = np.load('LORENZ_heatmap_100_data.npz', allow_pickle=True) # Iterate over all noise levels and outlier percentages for noise_key in data.keys(): if noise_key == 'noise_0': continue # Skip baseline noise_dict = data[noise_key] for data_key in noise_dict.keys(): if 'data_' in data_key: outlier_pct = float(data_key.split('_')[1]) noise_val = float(noise_key.split('_')[1]) realizations = noise_dict[data_key] # shape (100, 1001, n_states) outlier_indices = noise_dict[f'outInd_{outlier_pct}'] print(f"Noise: {noise_val}, Outlier%: {outlier_pct*100:.1f}%, " f"Realizations: {realizations.shape[0]}") Time Vector import numpy as np # SIR: time from 0 to 100 time_sir = np.linspace(0, 100, 1001) # Lorenz: time from 0 to 20 time_lorenz = np.linspace(0, 20, 1001) # Lotka-Volterra: time from 0 to 30 time_lv = np.linspace(0, 30, 1001) Data Statistics File Sizes (approximate) SIR_heatmap_100_data.npz: ~152 MB LORENZ_heatmap_100_data.npz: ~152 MB LV_heatmap_100_data.npz: ~103 MB (fewer states) Total Observations per File Time points: 1,001 Realizations: 100 Noise levels: 8 (+ 1 baseline) Outlier percentages: 8 Total unique parameter combinations: 9 × 8 = 72 Total time series: 72 × 100 = 7,200 per file Data Generation This dataset was generated using gen_heatmap_data.py, a Python utility that: Integrates the ODE system using scipy.integrate.solve_ivp Generates 1,001 time points for each system Creates 100 independent noise realizations for each noise/outlier combination Saves all realizations in a single compressed NPZ file Running the Generation Script # Generate SIR data with 100 realizations python gen_heatmap_data.py --case SIR --numSamples 100 -o ./data # Generate Lorenz data python gen_heatmap_data.py --case LORENZ --numSamples 100 -o ./data # Generate Lotka-Volterra data python gen_heatmap_data.py --case LV --numSamples 100 -o ./data Requirements: NumPy SciPy (for solve_ivp) Command-line arguments: --case: Model choice ('SIR', 'LORENZ', 'LV') --numSamples: Number of realizations per parameter combination (default: 100) -o, --outputDir: Output directory (default: '../data') Validation Total files: 3 Format consistency: All files use NPZ with pickle=True for dictionary compatibility Missing data: None Data integrity: All 100 realizations present for each parameter combination Recommended validation: import numpy as np for model in ['SIR', 'LORENZ', 'LV']: data = np.load(f'{model}_heatmap_100_data.npz', allow_pickle=True) # Check baseline exists assert 'noise_0' in data, f"Missing baseline for {model}" assert 'data_0' in data['noise_0'], f"Missing baseline data for {model}" # Check all noise levels for nl in [0.025, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2]: key = f'noise_{nl}' assert key in data, f"Missing noise level {nl} for {model}" assert len(data[key]) > 0, f"Empty data for noise {nl} in {model}"