Towards Reliable Generation of Executable Workflows by Foundation Models

This is the replication package for the manuscript, "Towards Reliable Generation of Executable Workflows by Foundation Models", published in ACM Transactions on Software Engineering and Methodology (TOSEM).The replication package includes workflows, generated by foundation models (FMs) as program instances of domain-specific languages (DSLs) and named as FM-generated DSL workflows, their open-coded labels, the detection and repair results of Timon and Pumbaa (our developed detection and repair tools), and also the templates of prompts utilized in designing Pumbaa.The replication package is structured as follows:- Detection: The directory containing the collective results of Timon, our static analysis detection tool for FM- generated DSL workflow programs (both simulated scenario-based workflows and OpenAGI workflows).- Repair: The directory containing the collective results of Pumbaa, our FM-based repair tool for FM-generated DSL workflow programs (both simulated scenario-based workflows and OpenAGI workflows).- Individual Reports (Detection + Repair per Workflow): The individual reports of Timon and Pumbaa for each workflow within the evaluation set.- Workflows: The directory including the open-coding instructions, the open-coded dataset, and the evaluation set used for investigating the performances of Timon and Pumbaa. Alongside of the workflows, the directory also includes the labels emerged from open-coding process. Additionally, the directory includes the workflows generated by Meta's Llama and Alibaba's Qwen, and their corresponding annotations, labeled for investigating the generalizability of the open-coding analysis (Section 3.3 of the manuscript). - Prompts: The directory includes the prompts templated utilized in the development of Pumbaa for repairing defect incidences within FM-generated DSL workflows.- README: The current file, describing the details on the replication package.## Note ##We are providing the discussed material as raw experiment results for examination; however, due to company policies, we cannot make the source code of our tools directly available. Meanwhile, we believe that we have provided sufficient implementation details in the manuscript for enabling reproducing our tools.