A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing

Overview This dataset is derived from the Big Sobol Sequence (BSQ) of the Quijote simulations, a collection of N-body simulations designed for machine learning applications. Each simulation consists of a point cloud (points in space, with 3D coordinates attached to them) generated under a varying set of cosmological parameters. Each point represents a simulated galaxy and is accompanied by associated properties such as velocity and mass. The cardinality of each point cloud is 5000 points. The dataset is split into 11,200 simulations in the training set, 608 in the validation set, and 576 in the test set. File structure The dataset is provided in TFRecord format. The training simulations are split across 50 TFRecord files following the naming convention halos_train_<i>.tfrecord. The validation and test sets are provided in halos_val_1.tfrecord and halos_test_1.tfrecord, respectively. Each dataset can be loaded using TensorFlow as shown in the code example below: import tensorflow as tffiles = tf.io.gfile.glob(f"halos*train*.tfrecord") # replace 'train' with 'val' or 'test' dataset = tf.data.TFRecordDataset(files) TFRecord structure Each record (corresponding to a point cloud) in a TFRecord file contains the following feature fields: "x": (tensor of shape [5000], dtype=float) - Position along x axis "y": (tensor of shape [5000], dtype=float) - Position along y axis "z": (tensor of shape [5000], dtype=float) - Position along z axis "v_x": (tensor of shape [5000], dtype=float) - Velocity along x axis "v_y": (tensor of shape [5000], dtype=float) - Velocity along y axis "v_z": (tensor of shape [5000], dtype=float) - Velocity along z axis "J_x": (tensor of shape [5000], dtype=float) - Angular momentum along x axis "J_y": (tensor of shape [5000], dtype=float) - Angular momentum along y axis "J_z": (tensor of shape [5000], dtype=float) - Angular momentum along z axis "M200c": (tensor of shape [5000], dtype=float) - Virial mass "Omega_m": (float) - Matter density "Omega_b": (float) - Baryon density "h": (float) - Hubble parameter "n_s": (float) - Density perturbation spectral index "sigma_8": (float) - RMS matter fluctuation amplitude on a scale of 8 Mpc/h "tpcf": (tensor of shape [24], dtype=float) - Two-point correlation function