HERITRACE user testing: packages and results

This dataset contains the results of the evaulation and the packages used to conduct the evaluation of HERITRACE, a semantic content management system for cultural heritage institutions. The study involved 14 participants divided into two groups: end users (7 participants) and configurators (7 participants).

Dataset contains:

Testing packages: two standalone Docker-based testing packages that enable independent reproduction and extension of the usability evaluation. 

User testing results: the dataset combines quantitative metrics (SUS scores, task duration, error rates, success rates) with qualitative analysis following grounded theory methodology (open, axial, and selective coding). All data is provided in accessible formats (JSON for structured data, Markdown for textual content, PNG for visualizations), enabling reproducibility and secondary analysis.

 Testing packages

  heritrace-enduser-testing.zip - End user testing package

Self-contained testing environment for metadata editors and domain experts

Pre-configured with 100 journal articles from OpenCitations Meta

Four guided tasks: editing records, merging duplicates, version restoration, and creating new entries

Duration: 60 minutes maximum

No Semantic Web knowledge required

  heritrace-technician-testing.zip - Configurator testing package

Environment for evaluating HERITRACE configuration capabilities

Two configuration tasks: SHACL schema extension and display rule customization

Tests system configurability for institutional customization

Duration: 60 minutes maximum

Semantic Web knowledge required

User testing results: directory structure

  heritrace_user_testing_results/  ├── aggregated_analysis/  │   ├── recruitment/ - Participant recruitment distribution data and visualizations  │   ├── sus/ - Aggregated System Usability Scale (SUS) reports, individual scores, and comparative visualizations  │   └── tasks/ - Cross-participant task metrics including duration comparisons, error heatmaps, and success rates  │  ├── endusers/ - Data from 7 end user participants (1e-7e)  │   ├── transcriptions/ - Complete verbatim transcriptions of testing sessions  │   ├── written_answers/ - Responses to post-test written questionnaires  │   ├── sus/ - Individual System Usability Scale questionnaires  │   ├── task_completion/ - Structured task performance data (completion status, duration, errors)  │   ├── open_coding/ - First-level qualitative coding of transcriptions  │   ├── end_user_axial_codes.json - Second-level categorical analysis  │   ├── end_user_selective_codes.json - Third-level theoretical integration  │   └── TEMPLATE_task_completion.json - Task completion data structure template  │  ├── technicians/ - Data from 7 technician/configurator participants (1t-7t)  │   ├── transcriptions/ - Complete verbatim transcriptions of testing sessions  │   ├── written_answers/ - Responses to post-test written questionnaires  │   ├── sus/ - Individual System Usability Scale questionnaires  │   ├── task_completion/ - Structured task performance data (completion status, duration, errors)  │   ├── open_coding/ - First-level qualitative coding of transcriptions  │   ├── technician_axial_codes.json - Second-level categorical analysis  │   ├── technician_selective_codes.json - Third-level theoretical integration  │   └── TEMPLATE_task_completion.json - Task completion data structure template  │  └── templates/ - Methodological templates documenting the analytical structure      ├── TEMPLATE_axial_codes.json - Axial coding structure      ├── TEMPLATE_open_codes.json - Open coding structure      └── TEMPLATE_selective_codes.json - Selective coding structure

本数据集包含面向文化遗产机构的语义内容管理系统HERITRACE的评估结果，以及用于开展该系统评估的相关工具包。本研究共招募14名参与者，分为两组：终端用户组（7名）与配置人员组（7名）。

数据集包含以下内容：

测试工具包：两款基于Docker的独立测试工具包，支持独立复现并拓展可用性评估流程。

用户测试结果：本数据集整合了定量指标（系统可用性量表（System Usability Scale, SUS）得分、任务时长、错误率、任务成功率）与基于扎根理论方法（开放编码、主轴编码、选择性编码）的定性分析结果。所有数据均采用可访问格式存储：结构化数据采用JSON格式、文本内容采用Markdown格式、可视化内容采用PNG格式，可支持研究复现与二次分析。

测试工具包
heritrace-enduser-testing.zip - 终端用户测试工具包
面向元数据编辑人员与领域专家的独立测试环境，预配置了来自OpenCitations Meta的100篇期刊文章。包含四项引导式任务：记录编辑、重复项合并、版本还原与新建条目。单轮测试最长时长为60分钟，无需具备语义网相关知识。

heritrace-technician-testing.zip - 配置人员测试工具包
该环境用于评估HERITRACE的配置能力，包含两项配置任务：SHACL模式扩展与展示规则自定义，用于测试系统面向机构定制的可配置性。单轮测试最长时长为60分钟，需具备语义网相关知识。

用户测试结果目录结构
heritrace_user_testing_results/
├── aggregated_analysis/
│ ├── recruitment/ - 参与者招募分布数据与可视化图表
│ ├── sus/ - 聚合版系统可用性量表（SUS）报告、个体得分与对比可视化图表
│ └── tasks/ - 跨参与者任务指标，包含时长对比、错误热图与任务成功率
│
├── endusers/ - 7名终端用户参与者（1e-7e）的相关数据
│ ├── transcriptions/ - 测试会话完整逐字转录文本
│ ├── written_answers/ - 测试后书面问卷的回复内容
│ ├── sus/ - 个体版系统可用性量表问卷
│ ├── task_completion/ - 结构化任务绩效数据（完成状态、时长、错误情况）
│ ├── open_coding/ - 转录文本的一级定性编码结果
│ ├── end_user_axial_codes.json - 二级分类分析结果
│ ├── end_user_selective_codes.json - 三级理论整合结果
│ └── TEMPLATE_task_completion.json - 任务完成数据结构模板
│
├── technicians/ - 7名技术人员/配置人员参与者（1t-7t）的相关数据
│ ├── transcriptions/ - 测试会话完整逐字转录文本
│ ├── written_answers/ - 测试后书面问卷的回复内容
│ ├── sus/ - 个体版系统可用性量表问卷
│ ├── task_completion/ - 结构化任务绩效数据（完成状态、时长、错误情况）
│ ├── open_coding/ - 转录文本的一级定性编码结果
│ ├── technician_axial_codes.json - 二级分类分析结果
│ ├── technician_selective_codes.json - 三级理论整合结果
│ └── TEMPLATE_task_completion.json - 任务完成数据结构模板
│
└── templates/ - 记录分析框架的方法论模板
├── TEMPLATE_axial_codes.json - 主轴编码结构模板
├── TEMPLATE_open_codes.json - 开放编码结构模板
└── TEMPLATE_selective_codes.json - 选择性编码结构模板