Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"

Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation" Journal: Science of Computer Programming Year: 2026 Authors: Eneko Pizarro, Maider Azanza, and Beatriz Perez This repository contains the replication package for the paper "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation". Repository structure 1) Test suite (resources under test) These are the classes and functions used for the analysis and metric collection in the paper. The table below lists each tested item, its origin, and the evaluation focus. Function/Method Function Type Evaluation Focus Source assemble Rental Assembler Complex object interactions, database mocking, multi-layered architecture https://github.com/BSPQ18-19/BSPQ19-S1.git isPrime Prime Checker Algorithmic logic, exception handling, high cyclomatic complexity, multiple error paths Software Engineering II (course), EHU/UPV addUser User Management Input validation, null handling, database mocking, CRUD operations https://github.com/jononekin/Forum-assertJ.git getBonus Bonus Calculator Nested conditional business rules, external dependencies https://github.com/jononekin/Forum-assertJ.git isIPV4Valid IPV4 Validator Utility function integration, helper function interaction https://www.geeksforgeeks.org/program-to-validate-an-ip-address/ isStrobogrammic Strobogrammatic Number Data structure manipulation (map/dictionary), low leakage risk https://www.w3resource.com/java-exercises/basic/java-basic-exercise-186.php palindrome Palindrome Detector Multi-function integration, coordinated test generation https://code-exercises.com/programming/hard/4/longest-palindrome-in-word isLeapYear Leap Year Checker Boundary conditions, multiple branching paths Software Engineering II (course), EHU/UPV 2) Aggregated results Folder: Aggregated_Results_by_Function_Oct.2025 Comprehensive results for all methods and LLMs tested. Data were collected between March 2024 and October 2025. The results include the metrics described in the evaluation framework section of the paper. Page index: Weights: Metric descriptions, calculation and evaluation methods, and aggregation weights. Journal-Summary Published: Aggregated results as reported in the paper. Comparison: GitHub test vs Custom Agent: Aggregated results comparing test generation methods (GitHub Copilot `/test` vs Custom Agent prompt). 3) Prompt artifacts Prompt_for_Integration_Testing-2025: Final prompt used to generate the test cases. Custom_Agent_Prompt_for_Integration_Testing.md: Custom Agent personalization structured prompt used in the measurement method. 4) Sources sources.7z: Source code classes and functions from which the test classes shown in the paper were generated. 5) Expert test cases Folder: Expert_Test_Cases Curated, human-authored test suites used as reference baselines for the evaluation. 6) Evaluation Folder: Evaluation Evaluation artifacts including: Evaluation_Consent_Form.pdf: Participant consent form used in the evaluation study. Evaluation_Information_Sheet.pdf`: Information sheet provided to evaluation participants. 7) Results and measurements Folder: Results_and_Measurements Raw and processed outputs from all measurement campaigns, organized by collection window and target function. 8) Related project Tetrics PoC: https://github.com/lksnext-ai-lab/tetrics.git Tetrics is a research prototype implementing a domain-independent, continuous evaluation framework for LLM-powered development tools.