Dataset for the paper: Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers

Dataset

This dataset contains the input and output examples used in the publication described below. The hyperspectral folder contains the first example and use case. The spatiotemporal folder contains the second example use case.

Contents

data/
├── hyperspectral
│   ├── VeloxTest-Membranes.emd
│   └── example_output
│       ├── simulator-0-VeloxTest-Membranes.emd
│       ├── simulator-0-VeloxTest-Membranes.json
│       └── simulator-0-VeloxTest-Membranes.png
└── spatiotemporal
    ├── 2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.emd
    ├── best-yolo.pt
    └── example_output
        ├── prediction-simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.mp4
        ├── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.emd
        ├── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.json
        └── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.mp4

Source Code
Our code is hosted on GitHub at the following link: https://github.com/ramanathanlab/PicoProbeDataFlow/tree/main

Paper Name
Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers

Paper Abstract
The Dynamic PicoProbe at Argonne National Laboratory is undergoing upgrades that will enable it to produce up to 100s of GB of data per day. While this data is highly important for both fundamental science and industrial applications, there is currently limited on-site infrastructure to handle these high-volume data streams. We address this problem by providing a software architecture capable of supporting large-scale data transfers to the neighboring supercomputers at the Argonne Leadership Computing Facility. To prepare for future scientific workflows, we implement two instructive use cases for hyperspectral and spatiotemporal datasets, which include: (i) off-site data transfer, (ii) machine learning/artificial intelligence and traditional data analysis approaches, and (iii) automatic metadata extraction and cataloging of experimental results. This infrastructure supports expected workloads and also provides domain scientists the ability to reinterrogate data from past experiments to yield additional scientific value and derive new insights.