Online Appendix - Making Sense of AI Agents Hype: Adoption, Architectures, and Takeaways from Practitioners

This replication package provides how we collect, extract, code, and analyze findings from Videos (transcripts) related to the AI agent-based.

In this research, the LLMs process was operated on Finnish Supercomputer using vLLM of LRM (Large Reasoning Models) and LLMs (Large Language Models).

Note: All the hyperlinks referring to the shared files only work in the local version (downloading the replication package). The only version will not work as the browser does not find the files.

License

All generated data is provided under DATA_LICENSE Creative Commons 4.0 Attribution License.

All scripts are provided under the Script_LICENSE MIT License.

Contents

This repository consists of the following files:

README.md: The README file describes the project and how to run the scripts to get the data and do the analysis.

INSTALL.md: Installation and configurations instructions.

requirements.txt: Library requirements for the Python environment.

Data_Collection

Analysis.xlsx: File to describe the process of the data selection. Among tabs, Tab ’Talks Selection’ describes selection of talks by application of the I/E criteria by 4 human assessors. Tab ‘I/E Criteria’ is the Inclusion and Exclusion Criteria used for the Talk Selection.

Scripts

00_Data_Collection

audioCutting.py: Script to cutting the audio.

audioExtraction.sh: Script to extract the audio.

transcriptExtraction.sh: Script to extract the transcripts.

videoDownload.sh: Script to download the videos.

Transcripts_Pipeline_Description: General description of the transcript extraction pipeline.

01_Extraction

01_videolinkscript_extraction.py: Script to extract evidence-grounded answers to 7 research questions from transcripts using retrieval-augmented generation and SWEBOK-aligned terminology refinement.

01_sbatch_extraction.sh: Script to submit the extraction work to the supercomputer.

02_Extraction_Validate

02_extraction_validate.py: Script to validate LLM-generated answers to research questions against transcript evidence using retrieval-augmented context and structured judgment prompts.

02_extraction_validate.sh: Script to submit the extraction validation work to the supercomputer.

03_ThematicAxial

03_thematicaxial_RQ1.py: Script to perform thematic and axial coding of RQ1-related excerpts by synthesizing timelines, motivations, and practitioner experiences into structured JSON output.

03_thematicaxial_RQ1.sh: Script to submit the corresponding RQ1 work to the supercomputer.

03_thematicaxial_RQ2.py: Script to perform thematic and axial coding of reported impacts of agent-based architectures (RQ2), labeled with supporting sources.

03_thematicaxial_RQ2.sh: Script to submit the corresponding RQ2 work to the supercomputer.

03_thematicaxial_RQ3.py: Script to perform thematic and axial coding of architectural considerations in agent-based migration or greenfield design (RQ3), labeled with supporting sources.

03_thematicaxial_RQ3.sh: Script to submit the corresponding RQ3 work to the supercomputer.

03_thematicaxial_RQ4.py: Script to perform thematic and axial coding of architectural strategies for agent selection and role assignment (RQ4), labeled with supporting sources.

03_thematicaxial_RQ4.sh: Script to submit the corresponding RQ4 work to the supercomputer.

03_thematicaxial_RQ5.py: Script to perform thematic and axial coding of architectural patterns for agent composition and coordination (RQ5), labeled with supporting sources.

03_thematicaxial_RQ5.sh: Script to submit the corresponding RQ5 work to the supercomputer.

03_thematicaxial_RQ6.py: Script to perform thematic and axial coding of domain- or task-specific differences in architectural practices (RQ6), labeled with supporting sources.

03_thematicaxial_RQ6.sh: Script to submit the corresponding RQ6 work to the supercomputer.

03_thematicaxial_RQ7.py: Script to perform thematic and axial coding of announced technologies used in LLM-based agentic systems (RQ7), labeled with supporting sources.

03_thematicaxial_RQ7.sh: Script to submit the corresponding RQ7 work to the supercomputer.

04_ThematicAxial_Validate

04_thematicaxial_RQ1Validate.py: Script to validate whether thematic and axial codes for RQ1 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq1.sh: Script to submit the corresponding RQ1 validation work to the supercomputer.

04_thematicaxial_RQ2Validate.py: Script to validate whether thematic and axial codes for RQ2 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq2.sh: Script to submit the corresponding RQ2 validation work to the supercomputer.

04_thematicaxial_RQ3Validate.py: Script to validate whether thematic and axial codes for RQ3 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq3.sh: Script to submit the corresponding RQ3 validation work to the supercomputer.

04_thematicaxial_RQ4Validate.py: Script to validate whether thematic and axial codes for RQ4 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq4.sh: Script to submit the corresponding RQ4 validation work to the supercomputer.

04_thematicaxial_RQ5Validate.py: Script to validate whether thematic and axial codes for RQ5 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq5.sh: Script to submit the corresponding RQ5 validation work to the supercomputer.

04_thematicaxial_RQ6Validate.py: Script to validate whether thematic and axial codes for RQ6 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq6.sh: Script to submit the corresponding RQ6 validation work to the supercomputer.

04_thematicaxial_RQ7Validate.py: Script to validate whether thematic and axial codes for RQ7 are correct and supported by the original source excerpts.

04_thematicaxial_validaterq7.sh: Script to submit the corresponding RQ7 validation work to the supercomputer.

Results

00_citation_groups_from_data_analysis.xlsx: citation groups (map talks IDs to GXX) from data analysis to the paper.

00_Data_Analysis.xlsx: Analysis of the generated results (including the whole results, and map 7 questions to three main objectives)

01_Extraction_Marco.csv: Raw results (extract relevant info for RQ) generated from Marco-o1.

02_ExtractionValidation_Llama.csv: Raw results (extraction validation - extract relevant info for RQ) generated from Llama.

02_ExtractionValidation_Mistral.csv: Raw results (extraction validation - extract relevant info for RQ) generated from Mistral Nemo.

02_ExtractionValidation_Qwen.csv: Raw results (extraction validation - extract relevant info for RQ) generated from Qwen.

03_ThematicAxial_Marco_RQ1.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ1 generated from Marco-o1.

03_ThematicAxial_Marco_RQ2.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ2 generated from Marco-o1.

03_ThematicAxial_Marco_RQ3.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ3 generated from Marco-o1.

03_ThematicAxial_Marco_RQ4.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ4 generated from Marco-o1.

03_ThematicAxial_Marco_RQ5.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ5 generated from Marco-o1.

03_ThematicAxial_Marco_RQ6.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ6 generated from Marco-o1.

03_ThematicAxial_Marco_RQ7.csv: Raw results (thematic+axial coding - classify and summarize the extraction results) regarding RQ7 generated from Marco-o1.

04_ThematicAxial_Validation_RQ1.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ1 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ2.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ2 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ3.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ3 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ4.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ4 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ5.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ5 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ6.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ6 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

04_ThematicAxial_Validation_RQ7.csv: Results (thematic+axial validation - classify and summarize the extraction results) regarding RQ7 generated from 3 LLMs (Mistral Nemo, Llama, and Qwen).

Comanions

Extended_Methodology.pdf: The extended Methodology of the paper that describe more details.

Linguistics_Metrics.pdf: The quantitative linguistic analysis of AI practitioner transcripts, focusing on length, lexical diversity, syntactic complexity, and domain specificity.

Linguistic_Metrics

linguistic_metrics.py: Script to extract linguistic and structural metrics from text corpora and export them in structured formats.

plots.py: Script to visualize and summarize the distributions of linguistic metrics through histograms, KDE curves, boxplots, and statistical summaries.

plots: Folder that store the results generated by plots.py

results: Folder that store the results generated by linguistic_metrics.py

transcripts: Folder that store all the video transcripts in our study

Replication of the results

This section describes the steps necessary to replicate the project, data selection, and using RAG and LLMs to perform data extraction and thematic+axial coding steps from our work.

Setting up the environment

Follow the instructions in INSTALL.md to set up the environment and define environment variables.

Data Collection

Analysis.xlsx describes the whole process of the data selection. Among tabs, Tab ’Talks Selection’ describes selection of talks by application of the I/E criteria by 4 human assessors. Tab ‘I/E Criteria’ is the Inclusion and Exclusion Criteria used for the Talk Selection.

The scripts of the data collection are shown in "Scripts/00_Data_Collection".

Data Extraction

LRM Running

After we get the above data (transcripts), we use it as the input for running LRM (Marco-o1) to do the data extraction.

We use 01_videolinkscript_extraction.py and 01_sbatch_extraction.sh, submit the work to the Finnish supercomputer, with the following command structure:

sbatch 01_sbatch_extraction.sh "MODEL_NAME" PORT OUTPUT_FILE

Then we get the results 01_Extraction_Marco.csv. Now the Data Extraction LRM Running process is completed.

Three LLMs Validation Running

After we get the results from LRM, we continue using 3 LLMs to validate the results. We use 02_extraction_validate.py and 02_extraction_validate.sh, submit the work to the supercomputer with the the following command structure:

sbatch 02_extraction_validate.sh "MODEL_NAME" PORT OUTPUT_FILE

Then we get the validation results from 3 LLMs:

02_ExtractionValidation_Llama.csv

02_ExtractionValidation_Mistral.csv

02_ExtractionValidation_Qwen.csv

Thematic+Axial Coding

LRM Running

After we get the results from data extraction, we split the results based on the RQs and try to do the thematic+axial coding for each RQ.

We use all the scripts in the [03_Thematic Axial](/Scripts/03_Thematic Axial) folder (7 scripts for 7 RQs and corresponding sbatch files), submit the work to the Finnish supercomputer, with the following command structure:

sbatch 03_thematicaxial_RQx.sh "MODEL_NAME" PORT OUTPUT_FILE

Then we get the results:

03_ThematicAxial_Marco_RQ1.csv

03_ThematicAxial_Marco_RQ2.csv

03_ThematicAxial_Marco_RQ3.csv

03_ThematicAxial_Marco_RQ4.csv

03_ThematicAxial_Marco_RQ5.csv

03_ThematicAxial_Marco_RQ6.csv

03_ThematicAxial_Marco_RQ7.csv

Now the Thematic+Axial Coding LRM Running process is completed.

Three LLMs Validation Running

After we get the results from LRM, we continue using 3 LLMs to validate the results. We use all the scripts in the [04_Thematic Axial](/Scripts/04_Thematic Axial_Validat) folder (7 scripts for 7 RQs validation and corresponding sbatch files), submit the work to the Finnish supercomputer, with the following command structure:

sbatch 04_thematicaxial_validaterq1.sh "MODEL_NAME" PORT OUTPUT_FILE RAW_LOG_FILE

Then we get the validation results from 3 LLMs:

04_ThematicAxial_Validation_RQ1.csv

04_ThematicAxial_Validation_RQ2.csv

04_ThematicAxial_Validation_RQ3.csv

04_ThematicAxial_Validation_RQ4.csv

04_ThematicAxial_Validation_RQ5.csv

04_ThematicAxial_Validation_RQ6.csv

04_ThematicAxial_Validation_RQ7.csv

Now the Thematic+Axial Coding Validation process is completed.

Data Analysis (RQs)

We do the data analysis based on the defined RQs. Then we map these 7 questions to three main objectives.

All the talks IDs were map to the citation groups (see [00_citation_groups from data analysis.xlsx](/Results/00_citation_groups from data analysis.xlsx))

All the process is shown in [00_Data Analysis.xlsx](/Results/00_Data Analysis.xlsx).

In addition, for the calculation and statistical analysis of linguistic metrics for video-to-text transcriptions, as well as the generation of corresponding results and visualizations, please refer to the Linguistic_Metrics folder.