LLM Gricean Maxims Interpretive and Generative Capabilities

Conversational applications of large language models (LLMs) require more than text generation in order to generate good user experiences in task-oriented domains such as customer service or reservations. This dataset supports research on the capability of LLMs to interpret and generate dialogue turns in accordance with (or in violation of) Gricean Maxims (Quantity, Quality, Relation, and Manner) defined by H. Paul Grice. The dataset is built around seed dialogues written by the authors, each representing a short human–chatbot exchange in a task-oriented scenario. Dialogues were designed either with no maxim violations or with violation of a specific maxim (explicit or subtle). Two tasks were evaluated across six LLMs (claude-opus-4-5-20251101, gpt-oss:120b, llama3.1:405b, llama3.2:3b, phi4:14b, vicuna:33b) using three prompt styles of increasing detail (A, B, C): - Interpretive task: LLMs evaluated the last turn of each seed dialogue and judged whether it violated any Gricean Maxim. Four human annotators performed the same task. - Generative task: LLMs generated a new last turn for the same predefined dialogue, either adhering to all maxims or violating a specific one. The resulting conversations were annotated by four human annotators. The dataset consists of three CSV files: LLM interpretations of seed dialogues (432 rows), human interpretations of seed dialogues (48 rows), and human annotations of LLM-generated dialogue turns (720 rows). All dialogues and annotations are in English. All prompts variations and dialogues are also included in .zip files.Funding This dataset was produced as part of the TrustBoost project (PID2023-150584OB-C22 and PID2023-150584OB-C21), funded by MICIU/AEI/10.13039/501100011033 and by the European Regional Development Fund (FEDER, UE); and the HORIZON EUROPE MSCA Staff Exchanges Action CRYSTAL (Grant no. 101182965).