Problems and countermeasures in natural language processing evaluation

NLP evaluation is an intuitive manifestation of the current technical level and research progress in NLP field, it also provides a benchmark and direction for the future development of related models and methods. Because of the guiding significance of NLP evaluation, it will seriously affect the fairness and objectivity of work evaluation in related fields if the evaluation itself has deviations.New evaluation data sets and tasks have been continuously proposed in recent years. However, the existing evaluation also exposed a series of problems in terms of scientificity and objectivity. Inappropriate evaluation will limit the advancement of natural language processing technology. This report will analyze the current situation and existing problems of NLP evaluation, as well as propose assumptions and prospects for future NLP evaluation.

自然语言处理（NLP）评估直观地展现了当前该领域的技术水平与科研进展，同时亦为相关模型与方法的未来发展提供了基准与导向。鉴于NLP评估的指导意义，若评估本身存在偏差，将严重影响到相关领域工作评价的公平性与客观性。近年来，新评价数据集与任务不断被提出。然而，现有评价亦暴露出一系列在科学性与客观性方面的问题。不恰当的评估将限制自然语言处理技术的进步。本报告将分析当前NLP评估的现状与存在的问题，并提出对未来NLP评估的假设与展望。