Online Appendix for "An Empirical Study on Agents for Microservice Architecture Reconstruction"

This Online Appendix accompanies the Registered Report:

An Empirical Study on Agents for Microservice Architecture Reconstruction — ICSME 2026.

The study investigates whether an LLM agent, equipped with a task specificationand a small procedural skill, can reconstruct the microservice architecture ofa software repository from its source code alone. This appendix provides thefull experimental materials that the Registered Report refers to but does notinline, so that every result reported in the paper can be inspected,re-executed, and reproduced from the artifacts in this folder.

Repository layout

VENLA RR/
├── README.md ← this file
├── Initial Task and Skill/
│ ├── TASK.MD ← the prompt given to the LLM agent
│ └── SKILL.md ← the procedural skill loaded alongside the prompt
└── LLM Selection/
├── README.md ← detailed description of the LLM selection procedure
├── artificialanalysis.csv ← input leaderboard snapshot (June 2026)
├── llmstats.csv ← input leaderboard snapshot (June 2026)
├── opencompass.csv ← input leaderboard snapshot (June 2026)
├── llm_selection.py ← rank-aggregation script
└── final_ranking.csv ← output: aggregated ranking, three selected LLMs at the top

What each folder contains

Initial Task and Skill/ — the agent setup

This folder contains the two artifacts that together define what theagent is asked to do and how it is steered while doing it. The pair isintentionally minimal: a short task specification plus a small proceduralskill, so that the agent's behaviour is reproducible and not hidden behindundocumented system prompts as it is only an INITIAL design. The finalimplementation of the paper will provide a refined task and a consolidated skill.

TASK.MD is the prompt given to the agent. It fixes the agent's role("automated architecture reconstruction tool"), defines the three conceptsit must extract from the repository (Service, Connection, Endpoint), andprescribes the exact output contract: a single raw JSON object with thekeys Services, Connections, and Endpoints, no commentary, no fences.

SKILL.md is a procedural skill loaded alongside the prompt. Itencodes the rules of thumb the agent should follow when mining therepository: how to detect service boundaries (Dockerfile, composeentries, deployment manifests, per-service entry points), how to findconnections (env URLs, depends_on, brokers, shared databases), and howto extract endpoints (HTTP route declarations, gRPC rpc, GraphQLfields, serverless event paths, Ingress resources). It also includes ashort pitfalls list — e.g. "do not infer connections from shared-libraryimports", "do not include HTTP methods in endpoint paths", "striphealth/metrics endpoints unless they are the only public surface".

Together, TASK.MD defines what the agent must produce and SKILL.mddefines how the agent should go about producing it.

LLM Selection/ — how the three study LLMs were chosen

Three LLMs were used in the study. They were selected by aggregating threepublic coding-task leaderboards (artificialanalysis.ai, llmstats.com,OpenCompass), all snapshotted in June 2026. The procedure is fullydeterministic and reproducible: a single Python script reads the threeinputs and writes a unified ranking to final_ranking.csv, from which thetop three entries were chosen.

For the full description of the aggregation procedure, the column schema,the median filter, the fuzzy name-matching, the three-way and pairwiseintersections, and the reproduction command, seeLLM Selection/README.md.

How the appendix maps onto the Registered Report

Registered Report element
Where it lives in this appendix

LLM agent task specification
Initial Task and Skill/TASK.MD

Procedural skill loaded by the agent
Initial Task and Skill/SKILL.md

Public leaderboards used to select the LLMs
LLM Selection/*.csv (three inputs)

LLM selection procedure
LLM Selection/llm_selection.py

Final aggregated ranking of candidate LLMs
LLM Selection/final_ranking.csv

The agent's actual outputs on the target repositories (the per-repoServices / Connections / Endpoints JSON objects, the ground truth,and the comparison metrics) are out of scope for this appendix and arereleased alongside the paper itself.

Reproducing the artifacts

Each folder is self-contained:

# Reproduce the LLM selection
cd "LLM Selection"
pip install pandas
python llm_selection.py
# → writes final_ranking.csv next to the script

The Initial Task and Skill/ folder is static text; no execution isrequired to inspect it.