Knowledge Graph Triple Validation by LLMs and Human-in-the-Loop

Suplementary material for the sumbitted article to the IPM Special issue on Large Language Models and Data Quality for Knowledge Graphs. The dataset is an extension of [1] and includes the following columns: subj the subject/head of the triple rel the predicate of the triple obj the object/tail of the triple support-level indicating the reliability of the triple ann-random[1: valid, 0: invalid], randomly selected annotation from the expert annotations avaialble in [1] ann-new [1: valid, 0: invalid], junior expert annotation gpt-4o-1 [1: valid, 0: invalid], response from 1st GPT prompt gpt-4o-2 [1: valid, 0: invalid], response from 2nd GPT prompt gpt-4o-3 [1: valid, 0: invalid], response from 3rd GPT prompt gpt-4o-majority [1: valid, 0: invalid], GPT annotation, computed as majority vote of gpt-4o-1,gpt-4o-2,gpt-4o-3 claude-1 [1: valid, 0: invalid], response from 1st claude prompt claude-2 [1: valid, 0: invalid], response from 2nd claude prompt claude-3 [1: valid, 0: invalid], response from 3rd claude prompt claude-majority [1: valid, 0: invalid], claude annotation, computed as majority vote of claude-1,claude-2,claude-3 llama-1 [1: valid, 0: invalid], response from 1st llama prompt llama-2 [1: valid, 0: invalid], response from 2nd llama prompt llama-3 [1: valid, 0: invalid], response from 3rd llama prompt llama-majority [1: valid, 0: invalid], llama annotation, computed as majority vote of llama-1,llama-2,llama-3 [1] https://github.com/danilo-dessi/SKG-pipeline/tree/main/eval