Order–Edit Interaction Diagnostics for Japanese Three-Valued Text Interpretation: SPTM v0.2 Reproducibility Dataset and Code

Description This repository provides the research data, experimental materials, and Python analysis code supporting the study “Order–Edit Interaction Diagnostics for Japanese Three-Valued Text Interpretation: An Exploratory Study of 13,160 Local-LLM Responses.” The study introduces Semantic Path and Threshold Mapping (SPTM), an exploratory framework for examining how controlled semantic edits alter the sensitivity of language-model interpretations to the order in which information is presented. Unlike conventional text analysis methods that primarily evaluate semantic similarity, classification accuracy, or response consistency, SPTM investigates interactions between information order and semantic modifications. Its primary objective is to characterize changes in interpretation stability and identify conditions under which order sensitivity emerges, disappears, or changes in magnitude. Dataset and Experimental Design The SPTM v0.2 dataset comprises: 80 synthetic Japanese text stimuli covering linguistic structures such as exceptions, negation, conditional rules, and evidential qualifications. 160 controlled text edits, including meaning-preserving and meaning-changing modifications. Six threshold-testing series for examining interpretation changes under controlled conditions. 13,160 recorded model responses, consisting of 6,580 trials each from Gemma 3 12B and DeepSeek-R1 14B. Three-valued interpretation labels: YES, NO, and UNKNOWN. The experimental framework systematically compares alternative information-presentation orders and semantic editing conditions. Human Validation and Reference Labels Two independent researchers conducted blinded linguistic assessments, followed by adjudication by a third researcher. The finalized reference dataset contains 80 accepted original stimuli and 150 accepted edits. Ten edits were excluded based on linguistic or experimental validity considerations. The final reference labels were established after the computational experiments and are used for exploratory reanalysis. Repository Contents The repository includes experimental stimuli, controlled edits, recorded model responses, finalized reference labels, Python analysis scripts, documentation, and integrity-verification resources. The materials support independent examination of the reported results, verification of analytical procedures, and further methodological development. Scope and Limitations This research uses synthetic Japanese texts and locally generated language-model outputs. It does not contain patient records or personal participant-level research data. The findings are exploratory and should not be interpreted as demonstrating universal superiority over conventional semantic-change detection methods. Instead, SPTM offers a complementary diagnostic perspective on the interaction between semantic editing and information-presentation order. Model-specific behavior, generation variability, and experimental design limitations should be considered when interpreting the results. Related Manuscript Harada, Y., & Miura, M. Order–Edit Interaction Diagnostics for Japanese Three-Valued Text Interpretation: An Exploratory Study of 13,160 Local-LLM Responses. Manuscript prepared for submission to Language Resources and Evaluation.