Current state of vulnerability handling from the perspective of software operators and developers: semi-structured interviews

This archive contains the transcripts of semi-structured interviews, conducted as part of the study "Leveraging Fine-grained Telemetry Data for the Detection and Prevention of Vulnerability Exploits" by Konrad Ponichtera and Sebastian Proksch. The goal of the interviews was to collect insights about the current state of vulnerability handling in the software engineering industry and identify improvement possibilities from the perspective of the system operator.

The interviews have been conducted with ten software engineers with system administration/operation backgrounds, meaning that either they perform system administration/operation as part of their responsibilities, or used to do so in the past. Each participant has been assigned an identifier from P1 to P10. Their roles and industry experience have been disclosed in the attached CSV file with the questionnaire responses.

Interview structure

The interviews were conducted online, and transcribed to the text file, which was then reviewed and pseudonymized. Each interview took approximately one hour. During each interview, the participants were asked to look into a problem of system vulnerabilities from the perspective of a system operator, who can configure and monitor the infrastructure, as well as deploy the applications. The participants were also informed that the operator cannot modify the applications' source code and artifacts. Then the participants were asked seven open questions about the status quo of vulnerability handling in software engineering and its three aspects of awareness, impact, and mitigation. The interviewees responded in accordance with their knowledge and experience.

Afterwards, the researcher guiding the session introduced the proposed system to the interviewees, by describing its goal and functionality. To avoid the moderator acceptance bias, we explained that the system was designed within the Software Engineering Research Group of Delft University of Technology. The mitigation of social desirability bias has been achieved by describing the actions taken by an imaginary operator named Albert.

After going through the wireframes of the operator's dashboard interface and describing the content of each screen, the participants were asked to follow a supervised walkthrough, where they played the role of an operator. They were presented with a hypothetical situation, where a critical vulnerability, similar to the Log4Shell appears on the dashboard. During this part, the guiding researcher was streaming the window of a wireframe editor. The interviewees were then asked to follow a think-aloud protocol, and indicate their actions as they "use" the system. Meanwhile, the researcher made changes to the wireframes to simulate the effect of the participants' actions.

After the walkthrough, participants filled out a System Usability Scale (SUS) questionnaire and were asked additional questions about the impact the proposed system would have on their vulnerability awareness, impact analysis, and mitigation capability.

Files

The archive contains three types of artifacts from the interview: pseudonymized transcripts, the questionnaire and the wireframes.

Transcripts

The transcripts directory contains ten text files with pseudonymized interview transcripts. In each file, the parts spoken by the interviewing researcher have been prepended with "R:", while the parts spoken by the interviewee have been prepended by the interviewee's identifier.

Questionnaire

The CSV file contains the answers participants gave at the start of the interview. Each participant was asked to provide their current job title and how many years they worked in the software engineering industry. They were also asked to specify which DevOps tools and technologies they used, and what system operator tasks they had a chance to do during their career. The goal of these questions was to estabilsh the interviewee's system operation background. Finally, after the prototype walkthrough, participants filled out SUS questionnaires, the results of which are included in the PDF.

Wireframes

The wireframes directory contains diagrams in DrawIO format (also exported to PDF), used during the interview.The vision GUI wireframes were used to introduce participants to what the envisioned system would look like.The walkthrough wireframes were used during the semi-supervised walkthrough scenario, where the interviewing researcher modified them following the participants' actions.