Paper2Fig100k dataset

Paper2Fig100k dataset A dataset with over 100k images of figures and text captions from research papers. Images of figures display diagrams, methodologies, and architectures of research papers in arXiv.org. We provide also text captions for each figure, and OCR detections and recognitions on the figures (bounding boxes and texts). The dataset structure consists of a directory called "figures" and two JSON files (train and test), that contain data from each figure. Each JSON object contains the following information about a figure: figure_id: Figure identification based on the arXiv identifier: <yymm>.<xxxxxx>-Figure<I>-<k>.png. captions: Text pairs extracted from the paper that relates to the figure. For instance, the actual caption of the figure or references to the figure in the manuscript. ocr_result: Result of performing OCR text recognition over the image. We provide a list of triplets (bounding box, confidence, text) present in the image. aspect: Aspect ratio of the image (H/W). Take a look at the OCR-VQGAN GitHub repository, which uses the Paper2Fig100k dataset to train an image encoder for figures and diagrams, that uses OCR perceptual loss to render clear and readable texts inside images. The dataset is explained in more detail in the paper OCR-VQGAN: Taming Text-within-Image Generation @WACV 2023 Paper abstract Synthetic image generation has recently experienced significant improvements in domains such as natural image or art generation. However, the problem of figure and diagram generation remains unexplored. A challenging aspect of generating figures and diagrams is effectively rendering readable texts within the images. To alleviate this problem, we present OCR-VQGAN, an image encoder, and decoder that leverages OCR pre-trained features to optimize a text perceptual loss, encouraging the architecture to preserve high-fidelity text and diagram structure. To explore our approach, we introduce the Paper2Fig100k dataset, with over 100k images of figures and texts from research papers. The figures show architecture diagrams and methodologies of articles available at arXiv.org from fields like artificial intelligence and computer vision. Figures usually include text and discrete objects, e.g., boxes in a diagram, with lines and arrows that connect them. We demonstrate the superiority of our method by conducting several experiments on the task of figure reconstruction. Additionally, we explore the qualitative and quantitative impact of weighting different perceptual metrics in the overall loss function.Paper2Fig100k 数据集（Paper2Fig100k dataset）是一个包含超过10万张来自学术论文的图表图片及文本标注的数据集。这些图表图片展示了arXiv.org平台上学术论文中的示意图、研究方法与架构图。本数据集同时提供每张图表的文本标注，以及针对图表的光学字符识别（Optical Character Recognition，OCR）检测与识别结果（包含边界框与识别文本）。 数据集的目录结构包含一个名为"figures"的文件夹，以及两个JSON文件（训练集与测试集），二者存储每张图表的相关数据。每个JSON对象包含某一图表的如下信息： 1. figure_id：基于arXiv标识符的图表编号，格式为<yymm>.<xxxxxx>-Figure<I>-<k>.png。 2. captions：从论文中提取的与该图表相关的文本对，例如图表的正式图注，或是论文手稿中对该图表的引用内容。 3. ocr_result：对该图片执行OCR文本识别得到的结果，以三元组列表形式提供，每个三元组包含（边界框、置信度、识别文本）。 4. aspect：图片的宽高比（H/W，即高度除以宽度）。 您可参考OCR-VQGAN（OCR-VQGAN）的GitHub仓库，该仓库利用Paper2Fig100k数据集训练面向图表与示意图的图像编码器，并采用OCR感知损失以在生成的图片中渲染清晰可读的文本。该数据集的详细说明可参阅论文《OCR-VQGAN: Taming Text-within-Image Generation》（发表于WACV 2023）。 ### 论文摘要 合成图像生成近年来在自然图像生成、艺术生成等领域取得了显著进展，但图表与示意图生成方向仍有待探索。生成图表与示意图的一大核心挑战在于如何高效渲染图像内部具备高可读性的文本。为解决这一问题，本文提出OCR-VQGAN模型——一种结合OCR预训练特征的编解码器模型，通过优化文本感知损失，促使模型保留高保真的文本与示意图结构。为验证所提方法的有效性，本文引入Paper2Fig100k数据集，该数据集包含超过10万张来自arXiv.org上学术论文的图表图片及文本，图表涵盖人工智能、计算机视觉等领域论文的架构示意图与研究方法图，通常包含文本与离散元素（如示意图中的方框）以及连接它们的线条与箭头。本文通过多项图表重建实验证明了所提方法的优越性，并探究了不同感知指标在整体损失函数中的权重对生成结果的定性与定量影响。