Knowledge Graph Triple Validation by LLMs and Human-in-the-Loop

Suplementary material for the sumbitted article to the IPM Special issue on Large Language Models and Data Quality for Knowledge Graphs. 

The dataset is an extension of [1] and includes the following columns:

subj the subject/head of the triple

rel the predicate of the triple

obj the object/tail of the triple

support-level indicating the reliability of the triple

ann-random[1: valid, 0: invalid], randomly selected annotation from the expert annotations avaialble in [1] 

ann-new [1: valid, 0: invalid], junior expert annotation 

gpt-4o-1 [1: valid, 0: invalid], response from 1st GPT prompt

gpt-4o-2 [1: valid, 0: invalid], response from 2nd GPT prompt

gpt-4o-3 [1: valid, 0: invalid], response from 3rd GPT prompt

gpt-4o-majority [1: valid, 0: invalid], GPT annotation, computed as majority vote of gpt-4o-1,gpt-4o-2,gpt-4o-3

claude-1 [1: valid, 0: invalid], response from 1st claude prompt

claude-2 [1: valid, 0: invalid], response from 2nd claude prompt

claude-3 [1: valid, 0: invalid], response from 3rd claude prompt

claude-majority [1: valid, 0: invalid], claude annotation, computed as majority vote of claude-1,claude-2,claude-3

llama-1 [1: valid, 0: invalid], response from 1st llama prompt

llama-2 [1: valid, 0: invalid], response from 2nd llama prompt

llama-3 [1: valid, 0: invalid], response from 3rd llama prompt

llama-majority [1: valid, 0: invalid], llama annotation, computed as majority vote of llama-1,llama-2,llama-3

[1] https://github.com/danilo-dessi/SKG-pipeline/tree/main/eval