A Fisher’s Exact Test Justification of the TF–IDF Term-Weighting Scheme

Term frequency–inverse document frequency, or TF–IDF for short, is arguably the most celebrated mathematical expression in the history of information retrieval. Conceived as a simple heuristic quantifying the extent to which a given term’s occurrences are concentrated in any one given document out of many, TF–IDF and its many variants are routinely used as term-weighting schemes in diverse text analysis applications. There is a growing body of scholarship dedicated to placing TF–IDF on a sound theoretical foundation. Building on that tradition, this article justifies the use of TF–IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective. We show that the common TF–IDF variant TF–ICF is, under mild regularity conditions, closely related to the negative logarithm of the <i>p</i>-value from a one-tailed version of Fisher’s exact test of statistical significance. As a corollary, we establish a connection between TF–IDF and the said negative log-transformed <i>p</i>-value under certain idealized assumptions. We further demonstrate, as a limiting case, that this same quantity converges to TF–IDF in the limit of an infinitely large document collection. The Fisher’s exact test justification of TF–IDF equips the working statistician with a ready explanation of the term-weighting scheme’s long-established effectiveness. Supplementary materials for this article are available online.

词频-逆文档频率（Term frequency–inverse document frequency，简称TF–IDF）可以说是信息检索发展史上最广为人知的数学表达式。其最初作为一种简单的启发式方法，用于量化某一特定词汇在众多文档中的单个文档内的出现集中程度，TF–IDF及其诸多变体如今已被广泛用作各类文本分析应用中的词汇加权方案。越来越多的学术研究致力于为TF–IDF建立坚实的理论基础。本文基于这一研究传统，通过展示如何从显著性检验的视角理解这一经典表达式，向统计学界论证了TF–IDF的合理性。我们证明，在温和的正则性条件下，常见的TF–IDF变体TF–ICF与费希尔精确检验（Fisher’s exact test）单侧版本所得p值（p-value）的负对数密切相关。作为推论，我们在特定理想化假设下，建立了TF–IDF与上述负对数变换p值之间的关联。我们进一步证明，作为一种极限情形，当文档集合无限大时，该统计量收敛于TF–IDF。基于费希尔精确检验的TF–IDF理论依据，为从业统计学家解释该词汇加权方案长期以来的有效性提供了现成的解释。本文的补充材料可在线获取。