Prompting Without Principles: Replication Package

# Prompting Without Principles: Are Students Transferring Software Engineering Knowledge to LLM Use? Generative AI (GenAI), particularly large language models (LLMs), are increasingly employed to assist software engineering activities. However, little attention has been paid to understanding how early career developers interact with these tools, especially software engineers with few years of experience but have learned software engineering discipline in the classroom, such as Master of Software Engineering Students. To investigate whether and how early career software engineers in masters classrooms transfer software engineering principles to their use of LLMs, we conducted a controlled study involving graduate students enrolled in a software engineering course titled Software Refactoring, offered in Fall 2024 as part of a top-tier Master of Software Engineering program. Our findings indicate that while students recognized the potential of use of LLMs, particularly its ability to accelerate certain tasks and scale, they often failed to apply disciplined software engineering practices when relying on the LLM. Our findings provide insights into the nature of software engineers' skills educators need to continuously reinforce as well as those new ones that need to be integrated when interacting with this new class of automated tools empowered by GenAI. ## We share replication artifacts of our experiments which include the following - Assignment text for ad-hoc experiment - "Manual Refactoring.docx"- Assignment text for automated experiment - "Automated Refactoring.docx"- Survey for ad-hoc experiment - "Traditional tools approach_Survey.csv"- Survey for automated experiment - "LLM Approach_Survey.csv"- Student prompts - Vysper - "vsyper-prompts-rep.xlsm" - Ari-toolkit - "ari-toolkit-prompts-rep.xlsm"- Prompt correctness analysis - "Quantitative Analysis - Manual versus Automated.xlsx"