Supplementary Material for: Concerns for the Reliability and Validity of the National Stroke Project Stroke Severity Scale

<i>Background:</i> The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of theNSP-SS. <i>Methods:</i> The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status. <i>Results:</i> The NSP-SS had moderate-substantial inter-rater (weighted kappa, ĸ_w = 0.66, 95% CI 0.55–0.77) and intra-rater (ĸ_w = 0.63, 95% CI 0.52–0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52–0.75, p &lt; 0.0001) but some categorizations in the NSP-SS seemed inappropriate reflecting poor content validity. Each NSP-SS point was associated with a greater likelihood of poor outcome (OR = 2.1, 95% CI 1.1–3.7, p = 0.016). Based on dichotomized scores (NSP 0–2 and NIH-SS &lt;6; mild deficits), the NSP-SS sensitivity was 70.9% (95% CI 57.9–81.2%), specificity 82.2% (95% CI 68.7–90.7%), likelihood ratio for severe stroke 4.0 (95% CI 2.1–7.6) and likelihood ratio for mild stroke 0.3 (95% CI 0.20–0.5). The dichotomized NSP-SS and NIH-SS similarly predicted poor outcome (NSP-SS &gt;2, OR = 4.7, 95% CI 1.7–13.0, p = 0.003 vs. NIH-SS ≧6, OR = 4.4, 95% CI 1.5–13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively). <i>Conclusion:</i> The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.

背景：国家卒中项目（National Stroke Project, NSP）是一项针对美国因卒中或短暂性脑缺血发作（transient ischemic attack, TIA）住院的医疗保险受益人群的回顾性队列研究。该项目包含一款卒中严重程度简易评估工具——NSP卒中量表（NSP-Stroke Scale, NSP-SS），其原本用于转归研究中的风险校正，但目前尚未有研究评估该量表的信度与效度。本研究旨在评估NSP-SS的信度、同时效度与结构效度。

方法：回顾性分析单家学术型医学中心内100例因缺血性卒中/TIA住院的连续性患者的初始神经系统检查资料。由同一名评估者对NSP-SS进行两次回顾性评分，并由第二名评估者独立评分，以评估其信度；同时回顾性评分美国国立卫生研究院卒中量表（National Institutes of Health Stroke Scale, NIH-SS），将其作为同时效度评估的金标准。结构效度则基于患者的出院转归状态进行分析。

结果：NSP-SS的评定者间加权Kappa值（ĸw=0.66，95%置信区间0.55~0.77）与评定者内加权Kappa值（ĸw=0.63，95%置信区间0.52~0.75）均体现出中等至极强的信度。NSP-SS与NIH-SS评分间的斯皮尔曼相关系数为0.65（95%置信区间0.52~0.75，p<0.0001），相关性中等，但NSP-SS的部分分类方式似乎并不合理，提示其内容效度欠佳。NSP-SS评分每增加1分，患者出现不良转归的比值比为2.1（95%置信区间1.1~3.7，p=0.016）。以二分法评分（NSP-SS评分0~2分且NIH-SS评分<6分定义为轻度神经功能缺损）为例，NSP-SS的灵敏度为70.9%（95%置信区间57.9%~81.2%），特异度为82.2%（95%置信区间68.7%~90.7%），重度卒中似然比为4.0（95%置信区间2.1~7.6），轻度卒中似然比为0.3（95%置信区间0.20~0.5）。二分法NSP-SS（评分>2分）与NIH-SS（评分≥6分）对不良转归的预测效能相当，二者的比值比分别为4.7（95%置信区间1.7~13.0，p=0.003）与4.4（95%置信区间1.5~13.0，p=0.006），且均具有极佳的区分度（C统计量分别为0.827与0.826）。

结论：相较于NIH-SS，NSP-SS具有中等至极强的信度，但内容效度欠佳，同时效度仅为差至中等水平。此外，目前尚不明确NSP-SS是否比NIH-SS更易于从病历中提取。综上，结合其其他局限性，该量表在未来卒中转归研究的风险校正中的应用价值仍有待商榷。