RDFS-LLM-Bench: A Benchmark for Evaluating RDF Schema Inference in LLMs

This dataset accompanies RDFS-LLM-Bench, a benchmark for systematicallyassessing the logical inference abilities of large language models (LLMs)with respect to RDFS entailment rules. The benchmark covers six RDFS entailment rules (rdfs2, rdfs3, rdfs5, rdfs7,rdfs9, rdfs11) and 19 entailment patterns (six 1-rule, seven 2-rule, andsix 3-rule patterns), seven dataset variants, and evaluation conditionsdefined by combinations of presented rule types (NRP/ARP) and rule formats(full/name/def). Contents:- lod-samples.zip: Raw SPARQL query results from DBpedia, Wikidata, and schema.org- datasets.zip: Benchmark datasets (seven variants: RK, LS, RVA, GS, GSC, NS, NSC)- validation.zip: Counterfactuality audit of the perturbed variants (LS, RVA, GS, GSC). Each pattern is checked for explicit presence in the source LOD; includes counterfactuality_summary.csv/xlsx, which reproduces the per-variant coverage and counterfactuality rates reported in the paper.- tasks.zip: Zero-shot prompt task files for each evaluation condition- requests.zip: LLM request files (OpenAI Batch / sequential format)- responses.zip: Raw LLM response files- eval.zip: Per-entry evaluation results (strict and flex modes)- reports.zip: Aggregated scores and composite metrics (CSV / Excel)- reasoning-trace-samples.zip: Reasoning trace samples collected separately from gpt-oss-120b and gpt-oss-20b. Because the main experimental pipeline did not capture reasoning content, a dedicated sampling pass was conducted specifically to record reasoning traces. Eight conditions are covered per model: NRP/ARP × full setting on RK/GS/NS, plus NRP/ARP × name setting on NS. Version 4.0.0:- Added counterfactuality validation results (validation.zip).- Flex-mode scoring now treats repeated identical output triples as a set, consistent with RDF graph semantics; evaluation results and reports were recomputed accordingly.- This version corresponds to the results reported in the accompanying paper. Data Sources:- DBpedia (https://dbpedia.org) — CC BY-SA 3.0- Wikidata (https://www.wikidata.org) — CC0 1.0- schema.org (https://schema.org) — CC BY-SA 3.0 Data was collected via SPARQL queries against the public endpoints of theabove sources.