odl-raiser/GGBench

--- license: mit language: - en tags: - generation - think-with-images - unified-multimodal-model pretty_name: GGBench task_categories: - visual-question-answering - text-to-image size_categories: - 1K<n<10K papers: - https://arxiv.org/abs/2511.11134 - https://huggingface.co/papers/2511.11134 arxiv: 2511.11134 --- ## Associated Paper This dataset is associated with the following paper: **GeoCraft / GGBench: A Comprehensive Benchmark for Geometry Construction** - arXiv: https://arxiv.org/abs/2511.11134 # GGBench Evaluation Script Documentation ## Directory Structure Overview ``` dataset/ ├── evaluate.py # Unified evaluation entry script ├── eval_prompts.py # Judge model prompt templates ├── GGBench_dataset.json # Official dataset (evaluation benchmark) ├── Q&A_image/ # Problem images and final result images ├── long_image/ # Long process images ├── eval_output/ # Default output directory for evaluation results └── requirements.txt # Python dependencies ``` ## 1. Data Download and Directory Setup 1. **Get the Dataset** ```bash git lfs install git clone https://huggingface.co/datasets/opendatalab-raiser/GGBench ``` 2. **Extract Dataset Archive** ```bash tar -xzvf dataset.tar.gz ``` After extraction, you will get the `dataset/` directory and original resource files. ## 2. Environment Setup 1. **Python Version**: Python 3.9 or higher is recommended. 2. **Install Dependencies**: ```bash pip install -r requirements.txt ``` ## 3. `evaluate.py` Configuration All configurable parameters are defined at the top of the script. Modify them according to your model and data paths before running: - `DATASET_PATH`: Path to GGBench dataset JSON, defaults to `./GGBench_dataset.json` in the current directory. - `MODEL_OUTPUT_PATH`: Path to the model output JSON (list structure) to be evaluated, default example is `test.json`. - `DATASET_ROOT` / `PRED_ROOT`: Root directories for original dataset resources and model-generated resources, used for resolving relative paths. - `OUTPUT_JSON` / `OUTPUT_JSONL`: Output locations for evaluation results. The script will automatically overwrite old entries and preserve other results. - `JUDGE_MODEL`, `JUDGE_URL`, `JUDGE_API_KEY`: Judge model name, base URL, and API key. The script uses OpenAI-compatible interface. - `MAX_WORKERS`: Number of concurrent threads. - `ENABLE_*` switches: Control whether to enable each evaluation module (final image judge, text chain judge, mid-process judge, LPIPS, PSNR, SSIM). - `LOG_FILE`: Log output location, defaults to `eval_output/evaluate.log`. ## 4. Input Data Requirements ### 4.1 GGBench Dataset (Ground Truth) - Located in `GGBench_dataset.json`, each sample contains fields such as `id`, `question`, `question_image`, `text_answer`, `res_image`, etc. - The script will automatically complete dataset information into model output items based on `id`. ### 4.2 Model Output File - The JSON pointed to by `MODEL_OUTPUT_PATH` needs to be a list, where each element contains: - `id`: Matches entries in the dataset. - Model-generated text fields (e.g., `output`) and image paths (e.g., `output_image_path` or `image_4`, etc.). - If there are intermediate process long images, provide `long_image_path`. - If paths are relative, they will be resolved based on `PRED_ROOT`. ## 5. Running the Script Execute in the `dataset/` directory: ```bash python evaluate.py ``` Script execution flow: 1. Initialize logging and dependencies. 2. Read GGBench dataset and model output, merge and complete information by `id`. 3. Execute each evaluation module according to switches: - **Final Image Judge** (`VLM_eval_image_result`): Calls judge model to compare reference image with model final image. - **Text Judge** (`eval_text_result`): Compares problem, reference answer with model text output. - **LPIPS / PSNR / SSIM**: Deep perception and pixel-level metrics, automatically handles fallback logic (missing images or exceptions return 0). - **Mid-Process Judge** (`Step Accuracy`, `Process Consistency`, `Problem-Solution Accuracy`): Evaluates multi-step generation process. 4. Write results to `OUTPUT_JSON` (list) and `OUTPUT_JSONL` (optional). If the target JSON already exists, the script will overwrite old records by `id`, and unevaluated entries will be preserved. After execution, you can check in the `eval_output/` directory: - `result.json`: Evaluation results summary. - `result.jsonl` (optional): Line-by-line JSON for streaming processing. - `score.json`: Aggregated total scores. - `evaluate.log`: Complete log including errors and warnings. ## 6. Common Scenarios and Recommendations - **Run Only Part of Modules**: Set the corresponding `ENABLE_*` constant to `False`, and the script will skip that evaluation. - **Batch Evaluation for Multiple Models**: Write an outer script to loop through modifying `MODEL_OUTPUT_PATH` and output paths, then call `python evaluate.py`. - **Judge Model Change**: Update `JUDGE_MODEL`, `JUDGE_URL`, `JUDGE_API_KEY`, and ensure the new model is compatible with `OpenAI`-style interface. - **Path Resolution Errors**: Check if `DATASET_ROOT` and `PRED_ROOT` are correct, ensure reference images and predicted images exist. When images are missing, the script will prompt in the log and set related metrics to 0. - **Incremental Evaluation**: Since the write logic overwrites by `id`, you can repeatedly run the script to update partial entries without manually cleaning old results. ## 7. Further Customization - Prompts are located in `eval_prompts.py`. If you need to adjust judge criteria or language, you can directly modify these templates. - To add new metrics, refer to existing function structures, add new modules in `evaluate.py` and enable them in the main flow. - If you want to save the complete output of the judge model, you can add `_raw_*` fields in each evaluation function or extend logging. --- If you encounter problems during use, first check `eval_output/evaluate.log` for error details, or debug with source code. Feel free to extend or integrate into larger evaluation workflows according to project needs. Happy evaluating!许可证：MIT 语言： - 英语 标签： - 生成任务 - 图像辅助思考（Think-with-Images） - 统一多模态模型（Unified Multimodal Model） 正式名称：GGBench 任务类别： - 视觉问答（Visual Question Answering） - 文本到图像生成（Text-to-Image Generation） 样本规模： - 1000 < 样本数 < 10000 关联论文： - https://arxiv.org/abs/2511.11134 - https://huggingface.co/papers/2511.11134 arXiv编号：2511.11134 --- ## 关联论文 本数据集关联以下论文： **GeoCraft / GGBench：面向几何构建的综合基准测试集** - arXiv: https://arxiv.org/abs/2511.11134 # GGBench 评估脚本文档 ## 目录结构概览 dataset/ ├── evaluate.py # 统一评估入口脚本 ├── eval_prompts.py # 评测模型提示词模板 ├── GGBench_dataset.json # 官方数据集（评估基准集） ├── Q&A_image/ # 问题图像与最终结果图像 ├── long_image/ # 长流程图像 ├── eval_output/ # 评估结果默认输出目录 └── requirements.txt # Python依赖项 ## 1. 数据集下载与目录配置 1. **获取数据集** bash git lfs install git clone https://huggingface.co/datasets/opendatalab-raiser/GGBench 2. **解压数据集归档文件** bash tar -xzvf dataset.tar.gz 解压完成后将得到`dataset/`目录与原始资源文件。 ## 2. 环境配置 1. **Python版本**：推荐使用Python 3.9及以上版本。 2. **安装依赖项**： bash pip install -r requirements.txt ## 3. `evaluate.py` 配置项 所有可配置参数均定义在脚本顶部，请在运行前根据您的模型与数据路径进行修改： - `DATASET_PATH`: GGBench数据集JSON文件路径，默认值为当前目录下的`./GGBench_dataset.json`。 - `MODEL_OUTPUT_PATH`: 待评估的模型输出JSON（列表结构）路径，默认示例为`test.json`。 - `DATASET_ROOT` / `PRED_ROOT`: 原始数据集资源与模型生成资源的根目录，用于解析相对路径。 - `OUTPUT_JSON` / `OUTPUT_JSONL`: 评估结果的输出位置。脚本将自动覆盖旧条目并保留其余结果。 - `JUDGE_MODEL`, `JUDGE_URL`, `JUDGE_API_KEY`: 评测模型名称、基础URL与API密钥。脚本采用兼容OpenAI的接口格式。 - `MAX_WORKERS`: 并发线程数。 - `ENABLE_*` 开关: 控制是否启用各评估模块（最终图像评测、文本链评测、中间流程评测、LPIPS、PSNR、SSIM）。 - `LOG_FILE`: 日志输出位置，默认值为`eval_output/evaluate.log`。 ## 4. 输入数据要求 ### 4.1 GGBench 数据集（真实标注集） - 数据集存储于`GGBench_dataset.json`中，每个样本包含`id`、`question`、`question_image`、`text_answer`、`res_image`等字段。 - 脚本将根据`id`自动将数据集信息补全至模型输出条目。 ### 4.2 模型输出文件 - `MODEL_OUTPUT_PATH`指向的JSON需为列表格式，每个元素包含： - `id`: 与数据集中的条目匹配。 - 模型生成的文本字段（例如`output`）与图像路径（例如`output_image_path`或`image_4`等）。 - 若存在中间流程长图像，请提供`long_image_path`字段。 - 若路径为相对路径，将基于`PRED_ROOT`进行解析。 ## 5. 脚本运行 在`dataset/`目录下执行以下命令： bash python evaluate.py 脚本执行流程： 1. 初始化日志与依赖项。 2. 读取GGBench数据集与模型输出，基于`id`合并并补全信息。 3. 根据开关配置执行各评估模块： - **最终图像评测**（`VLM_eval_image_result`）：调用评测模型对比参考图像与模型生成的最终图像。 - **文本评测**（`eval_text_result`）：对比问题、参考答案与模型的文本输出。 - **LPIPS / PSNR / SSIM**：深度感知与像素级评估指标，自动处理回退逻辑（图像缺失或异常时返回0）。 - **中间流程评测**（`Step Accuracy`, `Process Consistency`, `Problem-Solution Accuracy`）：评估多步骤生成流程。 4. 将结果写入`OUTPUT_JSON`（列表格式）与`OUTPUT_JSONL`（可选）。若目标JSON已存在，脚本将根据`id`覆盖旧记录，未评估条目将被保留。 执行完成后，可在`eval_output/`目录中查看以下文件： - `result.json`: 评估结果汇总。 - `result.jsonl` (optional): 用于流式处理的逐行JSON格式文件。 - `score.json`: 聚合后的总评分。 - `evaluate.log`: 包含错误与警告的完整日志。 ## 6. 常见场景与优化建议 - **仅运行部分模块**：将对应的`ENABLE_*`常量设为`False`，脚本将跳过该评估环节。 - **多模型批量评估**：编写外层脚本循环修改`MODEL_OUTPUT_PATH`与输出路径，随后调用`python evaluate.py`。 - **更换评测模型**：更新`JUDGE_MODEL`, `JUDGE_URL`, `JUDGE_API_KEY`，并确保新模型兼容OpenAI风格的接口。 - **路径解析错误**：检查`DATASET_ROOT`与`PRED_ROOT`是否正确，确保参考图像与预测图像存在。若图像缺失，脚本将在日志中提示并将相关指标设为0。 - **增量评估**：由于写入逻辑基于`id`覆盖旧数据，您可重复运行脚本以更新部分条目，无需手动清理旧结果。 ## 7. 进阶自定义 - 提示词存储于`eval_prompts.py`中。若需调整评测标准或语言，可直接修改这些模板。 - 新增评估指标：参考现有函数结构，在`evaluate.py`中新增模块并在主流程中启用。 - 保存评测模型完整输出：若需保存评测模型的完整输出，可在各评估函数中新增`_raw_*`字段或扩展日志功能。 --- 若使用过程中遇到问题，请首先查看`eval_output/evaluate.log`以获取错误详情，或通过源代码进行调试。您可根据项目需求自由扩展或集成至更大规模的评估工作流中。祝您评估顺利！