opendatalab/ScienceMetaBench

ScienceMetaBench是一个用于评估从科学文献PDF文件中提取元数据准确性的基准数据集。该数据集涵盖三大类别：学术论文、教科书和电子书，可用于评估视觉语言模型（VLMs）或其他信息提取系统的性能。数据集包含两批数据，共1010条记录，覆盖多种语言和学科。数据以JSONL格式存储，包含详细的元数据字段如DOI、ISBN、标题、作者等。评估基于字符串相似度算法，提供字段级准确率和整体准确率两个核心指标。数据集适用于LLM性能评估、信息提取系统测试、模型微调和跨语言能力评估等场景。

ScienceMetaBench is a benchmark dataset for evaluating the accuracy of metadata extraction from scientific literature PDF files. The dataset covers three major categories: academic papers, textbooks, and ebooks, and can be used to assess the performance of Vision Language Models (VLMs) or other information extraction systems. It includes two batches of data totaling 1010 records, covering multiple languages and disciplines. The data is stored in JSONL format with detailed metadata fields such as DOI, ISBN, title, author, etc. Evaluation is based on a string similarity algorithm, providing two core metrics: field-level accuracy and overall accuracy. The dataset is suitable for scenarios such as LLM performance evaluation, information extraction system testing, model fine-tuning, and cross-lingual capability evaluation.