ISMRM 2022/23 Reproducibility Team Challenge: Replication results for MReplIcators team

This repository contains the MReplIcators team’s replication attempt results from the 2022-23 International Society of Magnetic Resonance in Medicine (ISMRM) Challenge ‘Repeat It With Me: Reproducibility Team Challenge’ (ISMRM Challenge – ISMRM's Challenge Forums).

The challenge aims to “raise awareness of the importance of reproducible science through fostering collaboration between labs by reproducing ISMRM abstracts”.

The Reproducer sub-team was formed by Ebony R. Gunwhy (ERG) and Jemima Pilgrim-Morris (JPM) from POLARIS (POLARIS (Pulmonary, Lung and Respiratory Imaging Sheffield) | POLARIS | The University of Sheffield) at The University of Sheffield and the team worked on reproducing abstract #1700 "Sensitivity Analysis of the Bloch Equations" (ISMRM 2022) by Author sub-team Nick Scholand and Martin Uecker from The Institute of Medical Engineering at Graz University of Technology, Austria and Institute for Diagnostic and Interventional Radiology University Medical Center, Göttingen (1).

In this archive, a zipped results folder contains the independent results and notes from each member of the replicator sub-team (in ‘results/JPM’ and ‘results/ERG’, respectively). A more detailed overview of the abstract and replication (motivation, approach, challenges, and outcomes) can also be found in ‘Abstract replication details.pdf’. A shorter summary is provided here:

A successful replication ‘attempt’ was defined as…

Installation of provided software

Creation of simulations with the BART (2) simulation interface

A fully successful replication was defined as…

Successful recreation of the figures provided in the abstract

Examining goodness-of-fit by matching the provided ground truth dataset to some tolerances (normalised root mean square error; NRMSE)

Using quantitative measures of reproducibility (e.g., coefficient of variation) between original data computed by author sub-team and data computed by replicator sub-team

Application of technique to further sequences and tissue/sequence parameters

----------------------------------------------------------------------------

Contact between Replicator and Author sub-teams

----------------------------------------------------------------------------

An initial meeting was conducted to discuss how we should organise our participation as a team in this challenge. Some starting information was exchanged, in addition to the original abstract:

(A) Publicly published information about the studies:

(i) arXiv preprint of more detailed manuscript (2)

(ii) GitHub repository associated with the arXiv preprint, containing steps for reproducing the work relating to Figure 2 of the abstract (https://github.com/mrirecon/bloch-moba/tree/main/02_sens_analysis)

(iii) GitHub repositories with tutorials on using the BART reconstruction toolbox (https://github.com/mrirecon/bart-workshop)

(B) The author sub-team also created an additional GitHub repository made publicly available, containing interactive Google Colab and Binder notebooks introducing BART and its simulation framework, and providing a more step-by-step guide to allow for an exact replication of the work in the abstract (https://github.com/mrirecon/ismrm-2022-sensitivity-analysis-bloch-eq). The author sub-team also uploaded the ground truth dataset (their own simulation results, located within ‘02_irbssfp/ref/’ and ‘03_unprep_irbssfp/ref/) from the abstract to this repository.

In the first instance, the Replicator sub-team attempted a replication using only the publicly available information outlined in (A). Each member of the Replicator sub-team created their own GitHub forks of the repository listed in (A)(ii) to provide a starting point from which to document their replication from. These can be found at https://github.com/JemimaPM/bloch-moba-jpm (JPM) and https://github.com/EbonyGunwhy/bloch-moba-erg/tree/MReplIcators (ERG) and contain a commit history of all replication steps taken.

The additional repository described in (B) was only used to download the ground truth dataset for quantitatively assessing the replication results. This limitation was imposed to allow the Replicator sub-team autonomy in reproducing the work and to aid in gaining a more thorough understanding of the methodology involved. It was agreed that if required, the replicator sub-team would take help from the extra repository described in (B) and provide feedback to the authors afterwards summarising what degree of information was useful for replicating the work. Further interaction was limited to creating issues on GitHub and to quick queries via email, as needed.

----------------------------------------------------------------------------

Expected outcomes

----------------------------------------------------------------------------

The original study was based on numerical experiments. Therefore, the results were expected to be directly reproducible, however it was expected that possible differences may occur due to numerical noise. The major reason for this is the single floating-point arithmetic used in the provided BART software. Its presence can be tested by reproducing the study results on various machines multiple times. Numerical noise is expected to change between different hardware. Therefore, both members of the Replicator sub-team attempted to independently replicate the results on their own hardware. The used hardware in this challenge is specified below.

Replicators:

ERG: Intel(R) Core(TM) i5-1145G7 CPU @ 2.60GHz (4 cores, 8 logical processors)

JPM: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz  (4 cores, 8 logical processors)

Authors:

Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz (4 cores, 8 logical processors)

----------------------------------------------------------------------------

Replication results

----------------------------------------------------------------------------

Prior to starting the replication attempt, we agreed on the NRMSE tolerances below which we would define the replication as successful. These were NRMSE = 0.005 for the DQ derivative and NRMSE = 0.0001 for the SA derivative.

Due to time constraints, only a direct replication was performed. This was deemed successful as both replicators were able to reproduce the results within these tolerances. As expected, differences occurred between hardware, with 0% NMRSE achieved in the replication attempt of JPM. In addition, the reproduced figures were qualitatively compared to the abstract figures and these were found to agree.

Further notes and results from the Replicator sub-team attempts can be found within the ‘results’ folder for each respective replictor.

===========================================

Carbon footprint

This algorithm runs in 0.08 mins on 4 CPUs and on:

Intel i5-10400F: draws 572.04 Wh. Based in the United Kingdom, this has a carbon footprint of 132.21 g CO2e, which is equivalent to 0.14 tree-months (Green Algorithms i5-10400F (green-algorithms.org))

Intel i7-4930K: draws 1.10 kWh. Based in the United Kingdom, this has a carbon footprint of 253.13 g CO2e, which is equivalent to 0.28 tree-months (Green Algorithms i7-4930K (green-algorithms.org))

(calculated using green-algorithms.org v2.2 [1]. Note: at time of writing, the Green Algorithms calculator did not include our specific CPUs, therefore here we have calculated with similar processors. A request has been made on the Green Algorithms GitHub repository for our processors to be added in future).

===========================================

References

1. Scholand N, & Uecker M. Sensitivity Analysis of the Bloch Equations [abstract]. In: Proceedings of the 31st Annual Meeting of ISMRM, London, 2022. Abstract nr 1700.

2. Uecker M, Virtue P , Ong F, Murphy MJ, Alley MT, Vasanawala SS, Lustig M. Software Toolbox and Programming Library for Compressed Sensing and Parallel Imaging [abstract]. ISMRM Workshop on Data Sampling and Image Reconstruction, Sedona, 2013.

3. Scholand N, Wang X, Roeloffs V, Rosenzweig S, Uecker M. Quantitative Magnetic Resonance Imaging by Nonlinear Inversion of the Bloch Equations. arXiv:2209.08027 [Preprint]. 2022. doi: 10.48550/arXiv.2209.08027

4. Lannelongue L, Grealey J, Inouye M. Green Algorithms: Quantifying the Carbon Footprint of Computation. Adv. Sci. 2021;8(12):2100707. doi: 10.1002/advs.202100707