Supplementary Material for: Concerns for the Reliability and Validity of the National Stroke Project Stroke Severity Scale

Background: The National Stroke Project (NSP) was a retrospective cohort study of US Medicare beneficiaries hospitalized with stroke or transient ischemic attack (TIA). The NSP included a simple assessment of stroke severity (NSP-Stroke Scale, NSP-SS). Used for risk adjustment in outcome studies, the reliability and validity of the NSP-SS have not been assessed. We determined the reliability, concurrent and construct validity of theNSP-SS. Methods: The initial neurologic examinations of 100 consecutive patients hospitalized with ischemic stroke/TIA in a single academic medical center were reviewed. The NSP-SS was retrospectively scored twice by the same rater and independently by a second rater to assess reliability. The National Institutes of Health Stroke Scale (NIH-SS) was also scored retrospectively and used as the criterion standard for concurrent validity. Construct validity was based on discharge status. Results: The NSP-SS had moderate-substantial inter-rater (weighted kappa, ĸw = 0.66, 95% CI 0.55–0.77) and intra-rater (ĸw = 0.63, 95% CI 0.52–0.75) reliability. Correlation between NSP-SS and NIH-SS scores was moderate (Spearman r = 0.65, 95% CI 0.52–0.75, p 2, OR = 4.7, 95% CI 1.7–13.0, p = 0.003 vs. NIH-SS ≧6, OR = 4.4, 95% CI 1.5–13.0, p = 0.006) with excellent discrimination (C = 0.827 and 0.826, respectively). Conclusion: The NSP-SS has moderate-substantial reliability but poor content validity and poor to moderate concurrent validity as compared with the NIH-SS. In addition, it is not clear that the NSP-SS is easier to extract from medical records than the NIH-SS. Given this, and its other limitations, the utility of this scale for risk adjustment in future stroke outcome studies is questionable.