Surrogate Modeling Benchmark - 100D function

This dataset is related to the 100D function benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: https://uqworld.org/t/benchmark-case-100d-function/.

The experimental designs include datasets with 400, 800, 1200, 1600, and 2000 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses.

Description of the dataset file

The dataset file includes two variables:

ExpDesigns, and

ValidationSet.

Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case.

The structure can be summarized as follows:

ExpDesigns(i).X(j,k,l)

i: dataset group,

j: sample index,

k: variable index, and

l: replication index.

ExpDesigns(i).Y(j,m,l)

i, j, l: same as above,

m: computational model output index.

ValidationSet.X(j,k)

j, k: same as above.

ValidationSet.Y(j,m)

j, m: same as above.

Description of benchmarked metamodel competitors

The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity.

For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP).

Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters.

We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization.

For further details on the settings, please refer to the competitors.m file and UQLab user manuals:

S. Marelli, N. Luethen, B. Sudret, UQLab User Manual – Polynomial Chaos Expansions, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.

C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, UQLab User Manual – Kriging (Gaussian Process Modeling), Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.

R. Schoebi, S. Marelli, B. Sudret, UQLab User Manual – Polynomial Chaos Kriging, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.

Description of the results file

The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows:

Metrics.(CompetitorName)(i).(MetricName)(l)

i: dataset group,

l: replication index.

The description of the performance measures (metrics) can be found here: https://uqworld.org/t/metamodel-performance-measures/.

Additional files

We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language).

Acknowledgment

This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.