Supplemental material for 'Characterization of structure and mixing in nanoparticle hetero-aggregates using convolutional neural networks: 3D-reconstruction versus 2D-projection'

This is the supplemental data for the manuscript titled ‘Characterization of structure and mixing in nanoparticle hetero-aggregates using convolutional neural networks: 3D-reconstruction versus 2D-projection’ submitted to Ultramicroscopy.

Motivation:

Detection of nanoparticles and classification of the material type in scanning transmission electron microscopy (STEM) images can be a tedious task, if it has to be done manually. Therefore, a convolutional neural network (CNN) is trained to do this task for STEM-images of TiO2-WO3 nanoparticle hetero-aggregates. In conventional STEM, only 2D-projection images of the samples can be measured. STEM tomography allows for a 3D-reconstruction but it is a time-consuming and hence expensive task. In the present work, evaluations of 2D-projections are compared quantitatively to evaluations of 3D-reconstructions. For both evaluations a CNN is trained to predict particle positions and classify the material. The present dataset contains training and evaluation data and some code scripts that can be used after installation of the MMDetection toolbox (https://github.com/open-mmlab/mmdetection) to train the CNN for 2D-projection data. For 3D-reconstruction a StarDist-3D network is trained (https://github.com/stardist/stardist). Details are provided in the manuscript submitted to Ultramicroscopy and in the comments of the code scripts. For evaluation, we provide Python and MATLAB scripts.

Authors and funding:

The present dataset was created by the authors. The work was funded by the Deutsche Forschungsgemeinschaft within the priority program SPP2289 under contract numbers RO2057/17-1 and MA3333/25-1 and under contract number INST 144/462-1 FUGG.

Dataset description:

We provide several zip-archives. All of them contain two subfolders, one of them for 2D-projection data, the other one for 3D-reconstruction data.

training_data.zip contains the training data. In the 3D case, the subfolder mask contains the ground truth segmentation masks; the subfolder reconstruction contains the corresponding simulated 3D reconstructions. In the 2D case, the subfolder HAADF contains the 2D-projection images. In both cases, the subfolder json contains the annotation. Each file within the json folder provides for each image or reconstruction the following information:

·        aggregat_no: image id, the number of the corresponding image file

·        particle_position_x: list of particle position x-coordinates in nm

·        particle_position_y: list of particle position y-coordinates in nm

·        particle_position_z: list of particle position z-coordinates in nm

·        particle_radius: list of volume equivalent particle radii in nm

·        particle_type: list of particle types, 1: TiO2, 2: WO3

·        particle_shape: list of particle shapes: 0: sphere, 1: box, 2: icosahedron

·        rotation: list of particle rotations in rad. Each particle is rotated twice by the listed angle (before and after deformation)

·        deformation: list of particle deformations. After the first rotation the particle x-coordinates of the particle’s surface mesh are scaled by the factor listed in deformation, y- and z-coordinates are scaled according to 1/sqrt(deformation).

·        cluster_index: list of cluster indices for each particle

·        initial_cluster_index: list of initial cluster indices for each particle, before primary clusters of the same material were merged

·        fractal_dimension: the intended fractal dimension of the aggregate

·        fractal_dimension_true: the realized geometric fractal dimension of the aggregate (neglecting particle densities)

·        fractal_dimension_weight_true: the realized fractal dimension of the aggregate (including particle densities)

·        fractal_prefactor: fractal prefactor

·        mixing_ratio_intended: the intended mixing ratio (fraction of WO3 particles)

·        mixing_ratio_true: the realised mixing ratio (fraction of WO3 particles)

·        mixing_ratio_volume: the realised mixing ratio (fraction of WO3 volume)

·        mixing_ratio_weight: the realised mixing ratio (fraction of WO3 weight)

·        particle_1_rho: density of TiO2 used for the calculations

·        particle_1_size_mean: mean TiO2 radius

·        particle_1_size_min: smallest TiO2 radius

·        particle_1_size_max: largest TiO2 radius

·        particle_1_size_std: standard deviation of TiO2 radii

·        particle_1_clustersize: average TiO2 cluster size

·        particle_1_clustersize_init: average TiO2 cluster size of primary clusters (before merging into larger clusters)

·        particle_1_clustersize_init_intended: intended TiO2 cluster size of primary clusters

·        particle_2_rho: density of WO3 used for the calculations

·        particle_2_size_mean: mean WO3 radius

·        particle_2_size_min: smallest WO3 radius

·        particle_2_size_max: largest WO3 radius

·        particle_2_size_std: standard deviation of WO3 radii

·        particle_2_clustersize: average WO3 cluster size

·        particle_2_clustersize_init: average WO3 cluster size of primary clusters (before merging into larger clusters)

·        particle_2_clustersize_init_intended: intended WO3 cluster size of primary clusters

·        number_of_primary_particles: number of particles within the aggregate

·        gyration_radius_geometric: gyration radius of the aggregate (neglecting particle densities)

·        gyration_radius_weighted: gyration radius of the aggregate (including particle densities)

·        mean_coordination: mean total coordination number (particle contacts)

·        mean_coordination_heterogen: mean heterogeneous coordination number (contacts with particles of the different material)

·        mean_coordination_homogen: mean homogeneous coordination number (contacts with particles of the same material)

·        material_1: the name of the first material (TiO2)

·        material_2: the name of the second material (WO3)

·        radius_equiv: list of area equivalent particle radii (in projection) in nm

·        k_proj: projection direction of the aggregate: 0: z-direction (axis = 2), 1: x-direction (axis = 1), 2: y-direction (axis = 0)

·        polygons: list of polygons that surround the particle (COCO annotation)

·        bboxes: list of particle bounding boxes

·        aggregate_size: projected area of the aggregate translated into the radius of a circle in nm

·        n_pix: number of pixel per image in horizontal and vertical direction (squared images)

·        pixel_size: pixel size in nm

·        image_size: image size in nm

·        add_poisson_noise: 1 if poisson noise was added, 0 otherwise

·        frame_time: simulated frame time (required for poisson noise)

·        dwell_time: dwell time per pixel (required for poisson noise)

·        beam_current: beam current (required for poisson noise)

·        electrons_per_pixel: number of electrons per pixel

·        dose: electron dose in electrons per Å2

·        add_scan_noise: 1 if scan noise was added, 0 otherwise

·        beam_misposition: parameter that describes how far the beam can be misplaced in pm (required for scan noise)

·        scan_noise: parameter that describes how far the beam can be misplaced in pix (required for scan noise)

·        add_focus_dependence: 1 if a focus effect is included, 0 otherwise

·        add_partial_coating: 1 if some TiO2 particles were coated by a thin WO3 film, 0 otherwise.

·        data_format: data format of the images, e.g. uint8

For 3D reconstructions, the following information is added:

·        add_image_shifts, 1 if all images of the tilt series were shifted randomly by some pixel to account for misaligned images, 0 otherwise.

·        add_projection_noise: 1 if noise was added to projection angles, 0 otherwise.

·        N_SIRT: number of SIRT iterations for the 3D-reconstruction.

·        proj_angles: list of angles used for the projection directions of the tilt series in rad.

For the 2D case, there are 24000 training images, 5500 validation images, 5500 test images, and their corresponding annotations. Aggregates and STEM images were obtained with the algorithm explained in the main work. The important data for CNN training is extracted from the files of individual aggregates and concluded in the subfolder COCO. For training, validation and test data there is a file annotation_COCO.json that includes all information required for the CNN training.

For the 3D case, there are 100 simulated reconstructions that are divided into 80 training and 20 validation images within the training script.

The zip archive models.zip includes the two networks that were trained, evaluated and used for the investigation in the manuscript. In the 2D case, network weights are stored in the file 2D_projection/logs/fit/20240209-095952/iter_60000.pth. These weights can be loaded with the jupyter-notebook 2D_projection_prediction.ipynb. Furthermore, a configuration file, which is required by the notebooks, is stored as 2D_projection/logs/fit 20240209-095952/config_file.py. In the 3D case, network weights are stored in 3D_reconstruction/model/weights_best.h5. Also for this case, a configuration file is provided. The network can be loaded with the jupyter-notebook 3D_reconstruction_prediction.ipynb.

The zip archive experiment_measurement.zip includes the experimental 2D-projection images and the experimental 3D-reconstructions investigated in the manuscript. In the 3D case, we provide measured reconstructions as obtained by MATLAB and as transformed for the prediction with the CNN.

The zip archive experiment_prediction.zip includes predictions and visualizations obtained by the 2D and 3D CNNs.

The zip archive simulation_measurement.zip includes simulated 2D-projection images and simulated 3D-reconstructions that were not used for the network training. These simulations were used for evaluation of trained networks.

The zip archive simulation_prediction.zip includes predictions and visualizations obtained by the 2D and 3D CNNs for the simulated data that was not used during the training process.

In the zip archive code.zip, we provide several files with Python and MATLAB code that can be used for training, prediction and evaluation of the CNNs. A lot of information is provided within the comments and markdowns. For the application, it is required that the MMDetection toolbox and the StarDist3D framework are installed.

·        2D_projection_training.py: This Python script can be used for network training of the 2D Mask R-CNN after installation of the MMDetection toolbox.

·        3D_reconstruction_training.py: This Python script can be used for network training of the StarDist-3D network after installation of the StarDist package.

·        2D_projection_prediction.ipynb: This jupyter-notebook can be used for the application of a trained Mask R-CNN to experimental and simulated 2D-projection data.

·        3D_reconstruction_prediction.ipynb: This jupyter-notebook can be used for the application of a trained StarDist-3D network to experimental and simulated 3D-reconstruction data.

·        Evaluation_experiment.m: This MATLAB script is for the visualization and quantitative comparison of 2D and 3D experimental evaluations.

·        Evaluation_simulation.m: This MATLAB script is for the visualization and quantitative comparison of 2D and 3D evaluations of simulations.

·        particle_detection_functions.py: This Python script contains functions required by the jupyter-notebooks. Details can be found within the comments.

·        ASTRA_CM_plot_results.m: This MATLAB script provides functions for the visualization of experimental and simulated 3D-reconstructions.

·        ASTRA_CM_particle_detection.m: This MATLAB script is used for the quantitative evaluation of segmentations of 3D-reconstructions.

There is no confidential data in this dataset. It is neither offensive, nor insulting or threatening.

The dataset was generated to discriminate between TiO2 and WO3 nanoparticles in STEM-images and STEM tomography reconstructions. It might be possible that it can discriminate between different materials if the STEM contrast is similar to the contrast of TiO2 and WO3 but there is no guarantee.