KGCW 2024 Challenge @ ESWC 2024

Knowledge Graph Construction Workshop 2024: challenge

Knowledge graph construction of heterogeneous data has seen a lot of uptakein the last decade from compliance to performance optimizations with respectto execution time. Besides execution time as a metric for comparing knowledgegraph construction, other metrics e.g. CPU or memory usage are not considered.This challenge aims at benchmarking systems to find which RDF graphconstruction system optimizes for metrics e.g. execution time, CPU,memory usage, or a combination of these metrics.

Task description

The task is to reduce and report the execution time and computing resources(CPU and memory usage) for the parameters listed in this challenge, comparedto the state-of-the-art of the existing tools and the baseline results providedby this challenge. This challenge is not limited to execution times to createthe fastest pipeline, but also computing resources to achieve the most efficientpipeline.

We provide a tool which can execute such pipelines end-to-end. This tool alsocollects and aggregates the metrics such as execution time, CPU and memoryusage, necessary for this challenge as CSV files. Moreover, the informationabout the hardware used during the execution of the pipeline is available aswell to allow fairly comparing different pipelines. Your pipeline should consistof Docker images which can be executed on Linux to run the tool. The tool isalready tested with existing systems, relational databases e.g. MySQL andPostgreSQL, and triplestores e.g. Apache Jena Fuseki and OpenLink Virtuosowhich can be combined in any configuration. It is strongly encouraged to usethis tool for participating in this challenge. If you prefer to use a differenttool or our tool imposes technical requirements you cannot solve, please contactus directly.

Track 1: Conformance

The set of new specification for the RDF Mapping Language (RML) established by the W3C Community Group on Knowledge Graph Construction provide a set of test-cases for each module:

RML-Core

RML-IO

RML-CC

RML-FNML

RML-Star

These test-cases are evaluated in this Track of the Challenge to determine their feasibility, correctness, etc. by applying them in implementations. This Track is in Beta status because these new specifications have not seen any implementation yet, thus it may contain bugs and issues. If you find problems with the mappings, output, etc. please report them to the corresponding repository of each module.

Note: validating the output of the RML Star module automatically through the provided tooling is currently not possible, see https://github.com/kg-construct/challenge-tool/issues/1.

Through this Track we aim to spark development of  implementations for the new specifications and improve the test-cases. Let us know your problems with the test-cases and we will try to find a solution.

Track 2: Performance

Part 1: Knowledge Graph Construction Parameters

These parameters are evaluated using synthetic generated data to have moreinsights of their influence on the pipeline.

Data

Number of data records: scaling the data size vertically by the number of records with a fixed number of data properties (10K, 100K, 1M, 10M records).

Number of data properties: scaling the data size horizontally by the number of data properties with a fixed number of data records (1, 10, 20, 30 columns).

Number of duplicate values: scaling the number of duplicate values in the dataset (0%, 25%, 50%, 75%, 100%).

Number of empty values: scaling the number of empty values in the dataset (0%, 25%, 50%, 75%, 100%).

Number of input files: scaling the number of datasets (1, 5, 10, 15).

Mappings

Number of subjects: scaling the number of subjects with a fixed number of predicates and objects (1, 10, 20, 30 TMs).

Number of predicates and objects: scaling the number of predicates and objects with a fixed number of subjects (1, 10, 20, 30 POMs).

Number of and type of joins: scaling the number of joins and type of joins (1-1, N-1, 1-N, N-M)

Part 2: GTFS-Madrid-Bench

The GTFS-Madrid-Bench provides insights in the pipeline with real data from thepublic transport domain in Madrid.

Scaling

GTFS-1 SQL

GTFS-10 SQL

GTFS-100 SQL

GTFS-1000 SQL

Heterogeneity

GTFS-100 XML + JSON

GTFS-100 CSV + XML

GTFS-100 CSV + JSON

GTFS-100 SQL + XML + JSON + CSV

Example pipeline

The ground truth dataset and baseline results are generated in different stepsfor each parameter:

The provided CSV files and SQL schema are loaded into a MySQL relational database.

Mappings are executed by accessing the MySQL relational database to construct a knowledge graph in N-Triples as RDF format

The pipeline is executed 5 times from which the median execution time of eachstep is calculated and reported. Each step with the median execution time isthen reported in the baseline results with all its measured metrics.Knowledge graph construction timeout is set to 24 hours. The execution is performed with the following tool: https://github.com/kg-construct/challenge-tool,you can adapt the execution plans for this example pipeline to your own needs.

Each parameter has its own directory in the ground truth dataset with thefollowing files:

Input dataset as CSV.

Mapping file as RML.

Execution plan for the pipeline in metadata.json.

Datasets

Knowledge Graph Construction Parameters

The dataset consists of:

Input dataset as CSV for each parameter.

Mapping file as RML for each parameter.

Baseline results for each parameter with the example pipeline.

Ground truth dataset for each parameter generated with the example pipeline.

Format

All input datasets are provided as CSV, depending on the parameter that is beingevaluated, the number of rows and columns may differ. The first row is alwaysthe header of the CSV.

GTFS-Madrid-Bench

The dataset consists of:

Input dataset as CSV with SQL schema for the scaling and a combination of XML,

CSV, and JSON is provided for the heterogeneity.

Mapping file as RML for both scaling and heterogeneity.

SPARQL queries to retrieve the results.

Baseline results with the example pipeline.

Ground truth dataset generated with the example pipeline.

Format

CSV datasets always have a header as their first row.JSON and XML datasets have their own schema.

Evaluation criteria

Submissions must evaluate the following metrics:

Execution time of all the steps in the pipeline. The execution time of a step is the difference between the begin and end time of a step.

CPU time as the time spent in the CPU for all steps of the pipeline. The CPU time of a step is the difference between the begin and end CPU time of a step.

Minimal and maximal memory consumption for each step of the pipeline. The minimal and maximal memory consumption of a step is the minimum and maximum calculated of the memory consumption during the execution of a step.

Expected output

Duplicate values

Scale
Number of Triples

0 percent
2000000 triples

25 percent
1500020 triples 

50 percent
1000020 triples 

75 percent
500020 triples

100 percent
20 triples

Empty values

Scale
Number of Triples

0 percent
2000000 triples

25 percent
1500000 triples 

50 percent
1000000 triples 

75 percent
500000 triples

100 percent
0 triples

Mappings

Scale
Number of Triples

1TM + 15POM
1500000 triples

3TM + 5POM
1500000 triples 

5TM + 3POM 
1500000 triples 

15TM + 1POM
1500000 triples

Properties

Scale
Number of Triples

1M rows 1 column
1000000 triples

1M rows 10 columns
10000000 triples 

1M rows 20 columns
20000000 triples 

1M rows 30 columns
30000000 triples

Records

Scale
Number of Triples

10K rows 20 columns
200000 triples

100K rows 20 columns
2000000 triples 

1M rows 20 columns
20000000 triples 

10M rows 20 columns
200000000 triples

Joins

1-1 joins

Scale
Number of Triples

0 percent
0 triples

25 percent
125000 triples 

50 percent
250000 triples 

75 percent
375000 triples

100 percent
500000 triples

1-N joins

Scale
Number of Triples

1-10 0 percent
0 triples

1-10 25 percent
125000 triples 

1-10 50 percent
250000 triples 

1-10 75 percent
375000 triples

1-10 100 percent
500000 triples

1-5 50 percent
250000 triples

1-10 50 percent
250000 triples 

1-15 50 percent
250005 triples 

1-20 50 percent
250000 triples

1-N joins

Scale
Number of Triples

10-1 0 percent
0 triples

10-1 25 percent
125000 triples 

10-1 50 percent
250000 triples 

10-1 75 percent
375000 triples

10-1 100 percent
500000 triples

5-1 50 percent
250000 triples

10-1 50 percent
250000 triples 

15-1 50 percent
250005 triples 

20-1 50 percent
250000 triples

N-M joins

Scale
Number of Triples

5-5 50 percent
1374085 triples

10-5 50 percent
1375185 triples

5-10 50 percent 
1375290 triples

5-5 25 percent
718785 triples

5-5 50 percent
1374085 triples

5-5 75 percent 
1968100 triples

5-5 100 percent 
2500000 triples 

5-10 25 percent 
719310 triples

5-10 50 percent 
1375290 triples

5-10 75 percent 
1967660 triples

5-10 100 percent 
2500000 triples

10-5 25 percent 
719370 triples 

10-5 50 percent 
1375185 triples

10-5 75 percent 
1968235 triples

10-5 100 percent 
2500000 triples

GTFS Madrid Bench

Generated Knowledge Graph

Scale
Number of Triples

1
395953 triples

10
3959530 triples 

100
39595300 triples 

1000
395953000 triples

Queries

Query
Scale 1
Scale 10
Scale 100
Scale 1000

Q1
58540 results
585400 results
No results available
No results available

Q2
636 results
11998 results  
125565 results
1261368 results

Q3
421 results
4207 results 
42067 results
420667 results

Q4
13 results
130 results
1300 results
13000 results

Q5
35 results
350 results
3500 results
35000 results

Q6
1 result
1 result
1 result
1 result

Q7
68 results
67 results
67 results
53 results

Q8
35460 results
354600 results
No results available
No results available

Q9
130 results
1300 results
13000 results
130000 results

Q10
1 result
1 result
1 result
1 result

Q11
130 results
260 results 
260 results 
260 results 

Q12
13 results
130 results
1300 results
13000 results

Q13
265 results
2650 results
26500 results
265000 results

Q14
2234 results
22340 results
223400 results
No results available

Q15
592 results
8684 results
35502 results
206628 results

Q16
390 results
780 results
260 results
780 results

Q17
855 results
8550 results 
85500 results 
855000 results 

Q18
104 results
1300 results
13000 results
130000 results