test_bench

# Test Bench Public A benchmark for evaluating OCR and document understanding models on table extraction and chart data extraction from PDF documents. ## Purpose - Evaluate how well models extract structured data from PDF documents - Test table cell extraction accuracy (content + spatial relationships) - Test chart data point extraction accuracy - Provide reproducible, community-driven evaluation results ## Dataset Structure Each test case is a pair: a source **PDF** and a **test.json** file containing evaluation rules. ``` ├── chart_test/ │ └── default/ │ ├── *.pdf # Source PDF documents (charts) │ └── *.test.json # Evaluation rules (ground truth) ├── table_test/ │ └── default/ │ ├── *.pdf # Source PDF documents (tables) │ └── *.test.json # Evaluation rules (ground truth) ``` ## Test Categories ### Chart Tests Evaluates chart data extraction accuracy using `chart_data_point` rules. Each rule specifies: - **labels**: Axis labels identifying the data point (e.g., year, metric name) - **value**: Expected extracted value - **normalize_numbers**: Whether to normalize number formats before comparison ### Table Tests Evaluates table structure extraction using `table` rules that validate cell content and spatial relationships: - **cell**: The expected cell value to find in the extracted table - **up** / **down** / **left** / **right**: Expected adjacent cell content (directional neighbors) - **top_heading** / **left_heading**: Expected row/column heading for the cell ## Evaluation Each `.test.json` file contains paired evaluation rules for its corresponding PDF. The evaluation code compares model-generated markdown/text output against these rules to produce accuracy scores. ## Submitting Evaluation Results This dataset is registered as a **Benchmark** on Hugging Face. To submit evaluation results: 1. Run your OCR model on the PDF files in this dataset 2. Evaluate against the test.json rules using the evaluation code 3. Add a `.eval_results/*.yaml` file to your **model repo** with your scores: ```yaml # .eval_results/test-bench-public.yaml - dataset: id: llamaindex/test_bench # This benchmark's dataset ID task_id: table_test value: 85.3 # Your table test accuracy (%) - dataset: id: llamaindex/test_bench task_id: chart_test value: 78.1 # Your chart test accuracy (%) ``` Results will automatically appear on this benchmark's leaderboard and on your model page. ## Note This is a pipeline validation repo with minimal test data. The full benchmark will contain a larger and more diverse set of documents once the pipeline is confirmed working.