A Human-Rated Hospitality Review Benchmark for LLM- Generated Sentiment Quadruple Extraction

This dataset provides a human-rated benchmark for evaluating LLM-generated sentiment quadruples in hospitality reviews. It contains 40 hospitality reviews, 109 predicted–reference quadruple comparison pairs, gold reference annotations, LLM-generated outputs, exact-match F1 scores, Semantic-Aware Flexible Evaluation (SAFE) scores, and human ratings across three dimensions: output acceptability, semantic similarity to reference annotations, and perceived alignment with metric scoring behaviour. The benchmark supports evaluation of LLM-based Quad-ABSA systems, comparison of automatic evaluation metrics, analysis of exact-match versus semantic-aware scoring, and development of new metric baselines for structured sentiment analysis. Although originally constructed for validating SAFE, the dataset can be used independently to evaluate other LLM outputs and study cases where automatic metric scores diverge from human judgement.