BANCO v1.0.1: a stratified bank of fifteen synthetic clinical vignettes for the characterisation of multi-model AI systems
收藏资源简介:
This deposit holds the case material of the study preregistered at https://osf.io/c5qk7: a bank of fifteen clinical vignettes in Spanish, written to be read by a panel of language models the way a psychotherapist would read a colleague's case, together with the design record of the bank and the instrument and results of its validation by a panel of practising therapists. The vignettes are stratified along two crossed clinical dimensions. The first is the clinical picture, in four categories (depression, trauma, substance abuse and relational conflict), distributed as four, four, three and four vignettes. The second is the interpretive openness of the case, in three categories (a shared framework that reads the case ineffectively, a shared framework that reads it effectively, and a case that genuinely admits distinct framings), distributed as six, two and seven vignettes. The two dimensions cross into twelve cells, of which ten are populated; the two empty cells are documented in the design record. Each vignette runs to continuous prose of between 146 and 185 words, with a median of 159, and carries no diagnosis, aetiological hypothesis or treatment plan, so the framing is left to the reader. The vignettes are synthetic. They were written for this study and describe no real person. No clinical record was consulted, transcribed or adapted, and no vignette corresponds to an identifiable patient, either directly or as a composite. The fifteen vignettes were written by Lidia Mora-Valenciano, a practising psychotherapist, who holds the clinical criterion of the material. Mario Vega-Barbas coordinated the methodology and carried the material into the experimental system, without taking clinical decisions on the content. The clinical author took no part in the validation. Four practising therapists validated the bank, blind to one another and none of them a member of the research team. On the descriptive metrics that served as the primary reading, plausibility had a median of 4.5 across the bank and every vignette met the minimum acceptability threshold, while sufficiency had a median of 3.5 and thirteen of the fifteen met it. Agreement between raters was low: the intraclass correlation coefficient, in its two-way random, absolute-agreement, average-measures form, was 0.033 for plausibility and -0.127 for sufficiency, both in the poor range and with wide confidence intervals. The exercise establishes the plausibility and representativeness of the bank as characterisation material; it is not a psychometric validation. The bank was held fixed throughout the study, with no vignette added, withdrawn or edited after closure. Its identifying SHA-256 hash, ffd5d46b1c0f3a7efc105ab58ce22cc0c2c38c20d5eae03206270b2e0b52c1c2, is registered in the preregistration and recorded in the order plan of the executed runs, so any run of the study can be traced to the exact bank it was produced from. The hash is computed over a canonical form of the JSON rather than over the raw bytes; the procedure is given in the README. The bank is deposited exactly as it was sealed, so the file keeps its correspondence with the registered hash. It therefore carries working fields from the process by which it was built, including a per-case design rationale, planning marks for an extension that was not executed, and two template fields left unsubstituted at closure. The design rationales state the intention behind each vignette and the reading each framework should open, which is the judgement a study using this bank as a stimulus would be asking its readers to make on their own. Anyone using the bank as stimulus material should present the vignette text alone and keep the design rationale as documentation for the analyst. Access is restricted and granted on reasoned request, a condition set by the clinical authorship of the bank. The vignettes are teaching-grade clinical material, and open circulation would place them in general training corpora, which would compromise their use as stimuli in any later study of this kind. Requests should state who is asking, for what work, and what will be done with the material. Access is granted for research and teaching, and does not carry the right to redistribute the bank. Requests should be addressed to the corresponding author at mario.vega@upm.es. The deposit contains eleven files: the bank, the design record, the validation instrument, the briefing given to the validators, six files of validation results, and a README that documents all of them.



