Supplementary Dataset and Benchmark Logs: Multi-Vector Indexing Evaluation for System Dynamics Models

This repository contains the raw experimental datasets and query configurations generated for the empirical evaluation in the paper "Beyond Whole-Model Vectors: Graph-Aware Multi-Vector Representation for System Dynamics Models". The dataset was used to evaluate a novel graph-aware Multi-Vector Indexing architecture against a traditional Single-Vector baseline. The proposed methodology decomposes raw System Dynamics JSON graphs into distinct semantic segments (Stock-Flow Subsystems, Causal Paths, and Feedback Loops) to overcome the "Information Bottleneck" inherent in whole-model dense embeddings. Dataset Contents paper_a_multivector_benchmark.csv: Contains 1,701 unique query-to-model interactions derived from testing 63 diverse System Dynamics models against 27 complex, domain-specific conceptual queries. paper_a_group_analysis.csv: Details the semantic precision broken down by query scope (Specific/Structural vs. Broad/Conceptual). Benchmark Query Categorization To quantitatively investigate the structural drivers behind retrieval variance, the 27 conceptual queries were explicitly categorized into two distinct cohorts: Group A: Specific / Structural Queries (15 queries) (Targeting explicit causal paths, isolated feedback loops, or strict mathematical formulations) Causal relationship between lake water levels and groundwater reserves Herbivory influence on plant growth and water consumption Energy recovery processes in wastewater treatment plants Chemical usage and sludge accumulation in water treatment facilities Traffic flow optimization using signal control mechanisms Impact of project scheduling on ongoing construction tasks Smart building climate control and indoor temperature regulation Machine maintenance delays affecting raw material processing Feedback loops stabilizing bank liquidity during deposit withdrawals Inflation rates and their effect on money supply dynamics Cash flow accumulation in long-term investment portfolios Causal path from nutrient limitation to waste accumulation Feedback regulation mechanisms in stem cell differentiation Genetic mutation and selection effects on allele frequency Predator-prey dynamics using Rosenzweig MacArthur equations Group B: Broad / Conceptual Queries (12 queries) (Targeting macro-level model behaviors, general dynamics, and holistic resource management) Carbon dioxide absorption limits in ocean and forest ecosystems Water allocation trade-offs in crop irrigation systems General dynamics of urban growth and public infrastructure Urban growth models including unemployment dynamics and social infrastructure Passenger flow and congestion management in urban public transport District heating network efficiency and heat distribution Integration of renewable energy and battery storage in power grids Waste accumulation and recycling mechanisms in urban management Government budget allocation between taxation and public spending International trade balance and export growth mechanisms Consumer market dynamics involving supply, demand, and pricing Impact of infectious disease spread on susceptible populations Aggregate Benchmark Results (Console Summary) The execution logs confirm the semantic granularity gains and computational trade-offs associated with the multi-vector approach. Below is the aggregate summary from the benchmark execution: Evaluation Scope: Models evaluated: 63 Queries: 27 Total Interactions: 1,701 Avg Segments/Model: 2.1 Feedback Loops: 0.0 Causal Paths: 0.3 Stock-Flow Subs: 1.8 Retrieval Performance (MaxSim vs Baseline): Avg Single-Vector Cosine Similarity: 0.5627 Avg Multi-Vector Best Cosine Similarity: 0.5685 Avg Improvement (Δ): +0.0058 Models Improved: 961 out of 1,701 (56.5%) Performance by Query Category: Group A (Specific / Structural Queries): Multi-Vector Success (Δ > 0) in 605 out of 945 interactions (64.02%). Group B (Broad / Conceptual Queries): Multi-Vector Success (Δ > 0) in 356 out of 756 interactions (47.09%). Computational Latency Trade-off: Avg Decomposition Time: 0.14 ms Avg Single-Vector Embedding Time: 22,006.53 ms Avg Multi-Vector Embedding Time: 46,463.26 ms (all segments) Systemic Overhead Factor: 2.1x