Replication Package for "Pattern Hunters: Agents for Microservice API Pattern Detection"

This is the Replication package for the paper “Pattern Hunters: Agents for Microservice API Pattern Detection.” We investigated whether LLM-based coding agents can automatically detect Microservice API Patterns (MAPs) in open-source microservice systems. Starting from three published datasets, we selected 100 open-source microservice repositories and built a knowledge base of the 45 MAPs from the official pattern catalogue, together with a detection specification given to every agent as a reusable skill. Nine agent–LLM configurations — the coding agents Claude Code, OpenAI Codex and OpenCode, each combined with the LLMs GLM-5.2, Kimi-K2.7-Code and MiniMax-M3 served through Ollama Cloud — judged, for every repository and every pattern, whether the pattern is present. That is 4,500 repository–pattern pairs per configuration run, and the whole study was executed three times, giving 2,700 agent runs in total. We then measured the agreement between configurations, detection performance against 1,065 manually labelled pairs, and the consistency of the results across the three runs. This package contains everything needed to verify or extend that analysis: the repository selection trail, the knowledge base and the detection specification, the untouched output of all 2,700 agent runs, every intermediate and final result, and all analysis scripts. Licence Scripts are released under the MIT License; data files are released under CC BY 4.0. What is in the package DataCollection/ — the full selection trail from the 415 candidate projects of the three source datasets down to the 100 analysed repositories, including the language criteria and the clone URL and exact commit of every repository. DetectionSpec/ — the 45-pattern knowledge base (patterns_kb.json) and SKILL.md, the detection specification whose full text is appended verbatim to every prompt so that all nine configurations receive byte-identical instructions. GroundTruth/ — the 1,065 manually labelled repository–pattern pairs: three disjoint samples of 355, one per agent, each labelled independently by two authors and reviewed by others. RawResults/ — the untouched output of all 2,700 agent runs, with the detection result and the harness log for each run. Results/ — every intermediate and final output: the extracted judgement matrices for the three runs, and the tables and figures of RQ1 (agreement), RQ2 (detection performance) and RQ3 (consistency across runs). Script/ — the eight scripts that produce everything above: three detection harnesses (one per agent), the extraction script, the sample re-scoring script, and the three analysis scripts. README.md and INSTALL.md — full documentation of the package structure, the replication steps, and the setup of each component. How to use it All raw agent output is included, so every table and figure of the paper can be reproduced without the agents, without an Ollama account and without an internet connection. Six of the eight scripts use only the Python standard library, and the two that draw figures accept --no-figure. After unpacking the archive: mkdir -p work python3 Script/02_Extraction/extract_results.py --run 1 --base RawResults/raw_run1 --out work/run1 python3 Script/04_Analysis/rq1_agreement.py --matrix work/run1/matrix.csv --out work/rq1_run1 INSTALL.md gives the complete sequence for all three runs and all three research questions, as a single block that regenerates every result file. Re-running the detection itself instead requires the three agents installed locally, an Ollama Cloud account and API key, and the repositories cloned at their recorded commits; that setup, and the one path constant that must be filled in before the detection scripts run, are documented in INSTALL.md as well. The 100 analysed repositories are not redistributed here, because they are third-party projects under their own licences. DataCollection/03_final_100_repositories.csv records the clone URL and the exact commit of each one, so the states the agents analysed can be restored with git clone followed by git checkout <commit_sha>. Agents and models used Coding agents: Claude Code 2.1.278, OpenAI Codex CLI 0.153.4, OpenCode 1.18.31. LLMs: GLM-5.2, Kimi-K2.7-Code, MiniMax-M3. All three agents connected directly to Ollama Cloud, which served the three models; no local model serving was involved. Agents and LLMs are not deterministic, so re-running the detection produces slightly different results. Quantifying that variation is one of the research questions of the paper, and the package ships the output of all three executions so that it can be examined directly. Author and affiliation information is withheld while the paper is under double-blind review, and will be added to this record on acceptance.