Context-based Entity Recommendation on Real-Life Knowledge Work in Context (RLKWiC dataset)

RLKWiC Add-on: Benchmarking Dataset for Entity Recommendation

This benchmark, built on top of the Real-Life Knowledge Work in Context (RLKWiC) dataset, is designed to evaluate context-based entity recommendation by simulating a scenario where participants receive entities extracted from their activities across their defined contexts. 

In total, 1850 entity recommendations were generated across 56 contexts. After deduplication, these entities were presented to participants for explicit relevance assessment on a 3-point scale:

0 [Irrelevant]: Signifying a lack of relevance between the recommended entity and the context.

1 [Relevant] Denoting a connection between the entity and the context, although it may not fully represent it.

2 [Representative]: The entity closely aligns with the context, indicating a high level of relevance where the context can be inferred to be about this entity.

Participants could also suggest additional relevant entities. The resulting dataset comprises 1067 entities with explicit relevance scores, offering a resource for benchmarking entity recommendation in real-life knowledge work.