Supplementary Dataset and Benchmark Logs: Multi-Vector Indexing Evaluation for System Dynamics Models

This repository contains the raw experimental datasets and query configurations generated for the empirical evaluation in the paper "Beyond Whole-Model Vectors: Graph-Aware Multi-Vector Representation for System Dynamics Models". The dataset was used to evaluate a novel graph-aware Multi-Vector Indexing architecture against a traditional Single-Vector baseline. The proposed methodology decomposes raw System Dynamics JSON graphs into distinct semantic segments (Stock-Flow Subsystems, Causal Paths, and Feedback Loops) to overcome the "Information Bottleneck" inherent in whole-model dense embeddings. Dataset Contents paper_a_multivector_benchmark.csv: Contains 2,079 unique query-to-model interactions derived from testing 63 diverse System Dynamics models against 33 complex, domain-specific conceptual queries. paper_a_group_analysis.csv: Details the semantic precision broken down by query scope (Specific/Structural vs. Broad/Conceptual). paper_a_multivector_metrics.csv: Contains the Information Retrieval (IR) performance metrics (Precision@1, Precision@3, and Mean Reciprocal Rank - MRR), comparing the retrieval accuracy across the Multi-Vector, Single-Vector, and Naive-Chunk indexing strategies. Benchmark Query Categorization To quantitatively investigate the structural drivers behind retrieval variance, the 33 conceptual queries were explicitly categorized into two distinct cohorts: Group A: Specific / Structural Queries (18 queries) (Targeting explicit causal paths, isolated feedback loops, or strict mathematical formulations) Causal relationship between lake water levels and groundwater reserves Herbivory influence on plant growth and water consumption Energy recovery processes in wastewater treatment plants Chemical usage and sludge accumulation in water treatment facilities Traffic flow optimization using signal control mechanisms Impact of project scheduling on ongoing construction tasks Smart building climate control and indoor temperature regulation Machine maintenance delays affecting raw material processing Feedback loops stabilizing bank liquidity during deposit withdrawals Inflation rates and their effect on money supply dynamics Cash flow accumulation in long-term investment portfolios Causal path from nutrient limitation to waste accumulation Feedback regulation mechanisms in stem cell differentiation Genetic mutation and selection effects on allele frequency Predator-prey dynamics using Rosenzweig MacArthur equations Correlating urban passenger transport flow dynamics with residential housing growth patterns and smart climate regulation demands Modeling the structural transmission vectors of infectious disease spread through urban public transport passenger flows and indoor smart building climate systems Integrating urban signal control traffic optimization with district heating network distribution to enhance smart building indoor temperature regulation Group B: Broad / Conceptual Queries (15 queries) (Targeting macro-level model behaviors, general dynamics, and holistic resource management) Carbon dioxide absorption limits in ocean and forest ecosystems Water allocation trade-offs in crop irrigation systems General dynamics of urban growth and public infrastructure Urban growth models including unemployment dynamics and social infrastructure Passenger flow and congestion management in urban public transport District heating network efficiency and heat distribution Integration of renewable energy and battery storage in power grids Waste accumulation and recycling mechanisms in urban management Government budget allocation between taxation and public spending International trade balance and export growth mechanisms Consumer market dynamics involving supply, demand, and pricing Impact of infectious disease spread on susceptible populations How do infectious diseases in wildlife populations ripple through international trade balances and domestic inflation rates? How do shifts in ocean carbon dioxide absorption limits affect marine biological mutations and subsequently alter international trade balances for seafood exports? Correlating long-term investment portfolio cash flow accumulation with national inflation rates to optimize government budget taxation and public spending Aggregate Benchmark Results (Console Summary) The execution logs confirm the semantic granularity gains and computational trade-offs associated with the multi-vector approach. Below is the aggregate summary from the benchmark execution: Evaluation Scope: Models evaluated: 63 Queries: 33 Total Interactions: 2,079 Avg Segments/Model: 2.1 Feedback Loops: 0.0 Causal Paths: 0.3 Stock-Flow Subs: 1.8 Avg Naive Chunks/Model: 5.4 Retrieval Performance (MaxSim vs Baseline): Avg Single-Vector Cosine Similarity: 0.5618 Avg Multi-Vector Best Cosine Similarity: 0.5660 Avg Naive-Chunk Best Cosine Similarity: 0.4216 Avg Improvement (Δ): +0.0042 Models Improved (MV > SV): 1161 out of 2079 (55.8%) Models Improved (MV > NC): 2,053 out of 2,079 (98.7%) Information Retrieval (IR) Metrics: Single-Vec: P@1=0.8788, P@3=0.4343, MRR=0.9172 Multi-Vec: P@1=0.8788, P@3=0.4545, MRR=0.9182 (Gain vs SV: P@3 +0.0202, MRR +0.0010) Naive-Chunk: P@1=0.0909, P@3=0.0808, MRR=0.1902 Performance by Query Category: Group A (Specific / Structural Queries): Multi-Vector Success (Δ > 0) in 690 out of 1134 interactions (60.85%). Group B (Broad / Conceptual Queries): Multi-Vector Success (Δ > 0) in 471 out of 945 interactions (49.84%). Computational Latency Trade-off: Avg Decomposition Time: 0.17 ms Avg Single-Vector Embedding Time: 22,006.69 ms Avg Multi-Vector Embedding Time: 46,475.78 ms (all segments) Avg Naive Embedding: 118,904.27 ms (all chunks) Systemic Overhead Factor: 2.1x NC/SV Overhead: 5.4x