OntoLAMA: LAnguage Model Analysis for Ontology Subsumption Inference

About

OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference. The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively. The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts. Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic SI datasets and two Complex SI datasets.

Dataset Source
#Concepts
#EquivAxioms
#Datasets(Train/Dev/Test)

Schema.org
894
N/A

Atomic SI: 808/404/2, 830

DOID
11,157
N/A

Atomic SI: 90,500/11,312/11,314

FoodOn
30,995
2,383

Atomic SI: 768,486/96,060/96,062

Complex SI: 3,754/1,850/13,080

GO
43,303
11,456

Atomic SI: 772,870/96,608/96,610

Complex SI: 72,318/9,040/9,040

MNLI
N/A
N/A

biMNLI: 235,622/26,180/12,906

Citation

The relevant paper has been accepted at Findings of ACL 2023: https://aclanthology.org/2023.findings-acl.213/.

```@inproceedings{he2023language,
title={Language Model Analysis for Ontology Subsumption Inference},
author={He, Yuan and Chen, Jiaoyan and Jimenez-Ruiz, Ernesto and Dong, Hang and Horrocks, Ian},
booktitle={Findings of the Association for Computational Linguistics: ACL 2023},
pages={3439--3453},
year={2023}
}```

Links

See instructions at: https://krr-oxford.github.io/DeepOnto/ontolama/

We have made available a convenient access of these datasets through Huggingface: https://huggingface.co/datasets/krr-oxford/OntoLAMA

The arxiv version is available at: https://arxiv.org/abs/2302.06761

Contact

Yuan He (yuan.he(at)cs.ox.ac.uk)