timtkddn/wpml-eval-govdocs

WPML-Eval Govdocs是一个用于研究和评估的数据集，包含从Govdocs1公开语料库中提取的1,000个英文HTML文档及其对应的简化WordProcessingML（Office Open XML正文格式）参考文件。数据集按输入HTML的令牌长度分为10个桶，每个桶包含100个文档，以确保模型在不同上下文长度下的公平评估。数据集还包括原始MS Word文档作为参考。该数据集主要用于评估HTML到WordProcessingML的转换效果，重点关注文本和段落结构的保留。WPML-Eval Govdocs is a research/evaluation dataset consisting of **1,000 English HTML documents** extracted from the Govdocs1 public corpus and their corresponding simplified **WordProcessingML (Office Open XML body format)** reference files. The dataset is stratified into 10 buckets based on the token length of the input HTML (using tiktoken cl100k_base tokenizer), with 100 documents per bucket to ensure fair evaluation of models across different context lengths. Each bucket folder contains three subdirectories: input HTML, reference WordProcessingML, and original MS Word documents for reference. The dataset focuses on evaluating HTML to WordProcessingML conversion, particularly on preserving text and paragraph structures.