Code and data for "Executable simulation models help language models reason in conversations about dynamic systems"

This archive holds the code and data for the paper "Executable simulation models help language models reason in conversations about dynamic systems" The study tests whether language-model assistants can keep track of how a system will behave as facts about it arrive one message at a time. It includes 40 synthetic conversations, each generated from its own reference simulation, with an exact answer key after every message. Two language models (Claude Sonnet 5 and Claude Opus 5.5) answered them under four conditions: Claude Code without tools Claude Code with tools but with simulation blocked Claude Code with tools Stella Architect with its built-in assistant, Merlin Contents The conversations and their answer keys Every assistant's full output for every run, including the final questions asked again with all facts in one message The scripts and simulation files each condition produced, and the audits of them The conversations excluded from analysis, with the reason for each The hand-checked verdicts the analysis reads The analysis datasets, the figures, and the code that generates the conversations, runs the experiment, scores the replies and produces every number in the paper Reproducing the results Every reported number is computed by scripts from the raw outputs. The README gives the commands. They need Python 3.14 with pandas, NumPy, SciPy and Matplotlib, and Node 22 or later. The code is released under the MIT License and the data, text and figures under CC BY 4.0. sd-ai, the service Merlin runs on, is cited separately in the paper. Source repository: https://github.com/UB-IAD/dynamic-reasoning