Supplementary Material for: Concerns for the Reliability and Validity of the National Stroke Project Stroke Severity Scale

<i>Background:</i> The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of theNSP-SS. <i>Methods:</i> The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status. <i>Results:</i> The NSP-SS had moderate-substantial inter-rater (weighted kappa, ĸ_w = 0.66, 95% CI 0.55–0.77) and intra-rater (ĸ_w = 0.63, 95% CI 0.52–0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52–0.75, p &lt; 0.0001) but some categorizations in the NSP-SS seemed inappropriate reflecting poor content validity. Each NSP-SS point was associated with a greater likelihood of poor outcome (OR = 2.1, 95% CI 1.1–3.7, p = 0.016). Based on dichotomized scores (NSP 0–2 and NIH-SS &lt;6; mild deficits), the NSP-SS sensitivity was 70.9% (95% CI 57.9–81.2%), specificity 82.2% (95% CI 68.7–90.7%), likelihood ratio for severe stroke 4.0 (95% CI 2.1–7.6) and likelihood ratio for mild stroke 0.3 (95% CI 0.20–0.5). The dichotomized NSP-SS and NIH-SS similarly predicted poor outcome (NSP-SS &gt;2, OR = 4.7, 95% CI 1.7–13.0, p = 0.003 vs. NIH-SS ≧6, OR = 4.4, 95% CI 1.5–13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively). <i>Conclusion:</i> The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.

<i>背景：</i>国家卒中项目（National Stroke Project，NSP）是一项针对因卒中或短暂性脑缺血发作（transient ischemic attack，TIA）住院的美国医疗保险受益人的回顾性队列研究。该项目纳入了一项简易卒中严重程度评估工具——NSP卒中量表（NSP-Stroke Scale，NSP-SS）。该量表原本用于结局研究的风险校正，但目前尚未对其信度与效度进行评估。本研究旨在明确NSP-SS的信度、同时效度与结构效度。
<i>方法：</i>本研究回顾了某单学术型医疗中心内100例连续收治的缺血性卒中/TIA住院患者的初始神经系统检查记录。由同一名评估者对NSP-SS进行两次回顾性评分，同时由第二名评估者独立评分以评估其信度。同时对美国国立卫生研究院卒中量表（National Institutes of Health Stroke Scale，NIH-SS）进行回顾性评分，将其作为同时效度验证的效标标准。结构效度则基于患者出院状态进行分析。
<i>结果：</i>NSP-SS的评分者间加权Kappa系数（weighted kappa, κ_w = 0.66, 95% CI 0.55–0.77）与评分者内加权Kappa系数（κ_w = 0.63, 95% CI 0.52–0.75）均处于中等至较高水平。NSP-SS评分与NIH-SS评分间存在中等程度相关（Spearman相关系数r = 0.65, 95% CI 0.52–0.75, p < 0.0001），但NSP-SS的部分分类维度似乎存在不合理之处，提示其内容效度欠佳。每增加1分NSP-SS评分，患者预后不良的风险均显著升高（优势比（OR）= 2.1, 95% CI 1.1–3.7, p = 0.016）。以二分法评分（NSP-SS 0~2分与NIH-SS <6分定义为轻度神经功能缺损）为例，NSP-SS的灵敏度为70.9%（95% CI 57.9%~81.2%），特异度为82.2%（95% CI 68.7%~90.7%），重症卒中似然比为4.0（95% CI 2.1~7.6），轻症卒中似然比为0.3（95% CI 0.20~0.5）。二分法评分的NSP-SS与NIH-SS对不良预后的预测效能相当（NSP-SS >2分：OR = 4.7, 95% CI 1.7~13.0, p = 0.003；NIH-SS ≥6分：OR = 4.4, 95% CI 1.5~13.0, p = 0.006），二者均具有良好的区分能力（C指数分别为0.827与0.826）。
<i>结论：</i>相较于NIH-SS，NSP-SS具有中等至较高的信度，但内容效度欠佳，同时效度处于较差至中等水平。此外，目前尚无明确证据表明NSP-SS比NIH-SS更易于从医疗记录中提取评分。综合上述缺陷与其他局限性，该量表在未来卒中结局研究中用于风险校正的实用价值有待商榷。