FLEX Benchmark (False Presupposition Linguistic Evaluation eXperiment)

Citation:

If you use this dataset, please cite it as: Sieker, J., Lachenmaier, C., & Zarrieß, S. (2025). FLEX Benchmark (False Presupposition Linguistic Evaluation eXperiment) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15348857

The first two authors contributed equally to this work.

For a detailled description of the results of both experiments see:

On False Claims:Clara Lachenmaier, Judith Sieker, and Sina Zarrieß. 2025. Can LLMs Ground when they (Don’t) Know: A Study on Direct and Loaded Political Questions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14956–14975, Vienna, Austria. Association for Computational Linguistics.

On False Scenarios: Sieker, J., Lachenmaier, C., & Zarrieß, S. (2025). LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High. Proceedings of the Annual Meeting of the Cognitive Science Society, 47. 

Description:

The FLEX Benchmark investigates whether Large Language Models (LLMs) can correctly reject false presuppositions or whether they incorrectly accommodate them. The dataset consists of two experiments, False Claims and False Scenarios. Both datasets focus on prompts situated in the political domain, i.e. they evaluate LLMs' ability to handle false presuppositions in scenarios where misinformation can have significant consequences. 

In both experiments, each question was presented to the LLMs three times. Subsequently, human annotators labeled the models' responses. 

A: Misinformation Accommodated

U: Imprecise Answer

N: Misinformation Rejected

The dataset is organized as follows:

`FLEX_claims_direct.csv`: Direct confirmatory and disconfirmatory questions from the False Claims Experiment  

`FLEX_claims_loaded.csv`: Loaded questions from the False Claims Experiment  

`FLEX_scenarios.csv`: Loaded questions from the False Scenarios Experiment  

The FLEX benchmark comprises 20,520 manually annotated data points in total, with 10,584 from the False Claims experiment and 9,936 from the False Scenarios experiment.

False Claims ExperimentThis experiment investigates the models’ capacity for communicative grounding  by comparing their responses to questions embedding false presuppositions and to direct factual questions. 

Source: Wahl-O-Mat (German Federal Agency for Political Education)  

38 Thirty-eight statements (S) regarding the positions of four major German parties (DIE LINKE, AfD, SPD, CDU/CSU) on issues like currency replacement and border controls incorporated into different question formats.

Direct confirmatory: 'Is it true that S?' 

Direct disconformatory: 'Is it true that not S?' 

Loaded: Does X know that S?  

Data Format: both CSV-Files share the following columns: 

Party: The party the question was about (e.g. DIE LINKE).

Topic: The overall topic (e.g. EU-Steuern).

Prompt: The polar question asked to the LLM 3 times.

{model}_{#}: The answer of model {model} of prompting iteration {#}.

{model}_{#}_annotation: Corresponding human Annotation to {model}'s answer No. {#}.

Flex_claims_direct comes with one additional column: 

Information: Contains FALSE if the question is disconfirmatory and TRUE if the question is confirmatory.

False Scenarios ExperimentThis experiment investigates how linguistic features (such as presupposition trigger type, embedding contexts, or plausibility) influence models’ susceptibility to false presuppositions. 

Source: Manually generated false assumptions regarding political events and actions, framed as polar questions. each prompt falsely associates a politician from one party with the congress of another party

Data Format:

prompt: The polar question asked to the LLM 3 times

triggertype: Contains the type of presupposition trigger (factive verb, change of state verb, Interaction particle, possesive, quantifier, temporal adjunct or temporal clause)

trigger: Contains the actual presupposition trigger

scenario_condition: Provides information about the political alignment of the person and the party congress related to the presupposition (e.g., a left-wing politician at a right-wing party congress would be "left_in_right").

parties_involved: Lists the actual parties of both the person and the party congress.

probability: Indicates the likelihood of the invented scenario in the context of a party congress (e.g., giving a speech: high(er); organizing a game night: low).

presupposes: Contains solely the presupposed content.

context: Specifies one of three embedding contexts (question, negation, modal).

{model}_{#}: The answer of model {model} of prompting iteration {#}.

{model}_{#}_annotation: Corresponding human Annotation to {model}'s answer No. {#}.