DesignSafe Annual Evaluation Report 2023-2024

This report summarizes the 2022-23 evaluation of the DesignSafe cyberinfrastructure, which was conducted by TACC’s Expanding Pathways in Computing (EPIC) STEM Evaluation Services. EPIC used a mixed methods approach to assess the implementation, outcomes, and impacts of DesignSafe on the natural hazards community. The evaluation aims to inform project improvements and report on outcomes over the project’s duration. Multiple data sources were used, including platform analytics, user profiles, annual and microsurvey responses, trainee surveys, and user and stakeholder interviews.

Annual user survey findings indicated users have high overall satisfaction with DesignSafe and believe it to be useful to individuals in the natural hazards community. Respondents reused data for research purposes, to analyze other researchers’ data, to collaborate with others, and to publish, present, or propose research. Meanwhile, users reported challenges including glitches and the complexity of DesignSafe and its various components. Microsurvey respondents indicated they download data from the Data Depot primarily for education purposes. Training participants were engaged in and satisfied with the trainings, and they planned to apply their learning. They suggested the trainings include more examples and be extended in length. User interviews provided insights in how DesignSafe facilitated users’ research and collaboration. User interviewees expressed interest in trainings and more customized functions and applications. Stakeholders discussed DesignSafe’s major accomplishments including publishing the HPC Jupyter, moving to Tapis V3, and improving the platform and its supports. Stakeholder interviewees recommended focusing efforts on AI/ML and computing resources and improving documentation, training, outreach, use cases, and user experiences. Based on these findings, the evaluators recommend DesignSafe resolve technical glitches, improve access to technical resources, provide more user support for and consider changes to the Data Depot, enhance the user experience and site navigation, educate users and potential users, and pursue next steps strategically.