From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data [Replication Package]

Artifact Summary This repository contains the replication package for the paper "From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data," accepted at the 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME'26). The purpose of the package is to facilitate the verification and reproduction of the study results. It provides artifacts for all seven research activities. Paper Abstract Scientists at the European XFEL conduct experiments that generate very large and complex datasets. The subsequent data analysis is challenging as scientists must combine their domain expertise with facility- and software-specific knowledge scattered across documentation, tools, and support channels. To address this problem, we designed and evaluated an agentic artificial intelligence (AI) system tailored to the scientists’ needs and integrated with the European XFEL high-performance computing environment. Using a design science research approach, we conducted a rapid literature review, a systematic evaluation of 16 AI tools, multiple interviews, a focus group, and a user study with experts at European XFEL to develop and evaluate two prototypes. Our study identifies key knowledge challenges in scientific data analysis, derives requirements for an AI agent that supports knowledge retrieval and code generation, and proposes design recommendations for a specialized system that is adaptable to the evolving AI tool landscape. Our findings provide guidance for developing maintainable AI support in highly specialized scientific environments. References The published paper will be available on [IEEE Xplore](/) and the preprint on [arXiv](/). // TODO add links to Xplore and arXiv